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I.  INTRODUCTION  AND  BACKGROUND 

A.  MOTIVATION 

Software  metrics  is  the  buzzword  today  in  both 
development  and  maintenance  activities.  Process  metrics 
focus  on  the  activities  involved  in  software  development  or 
maintenance .  The  product  metric  focuses  on  individual 
aspects  of  the  item  (usually  volume)  under  development  or 
maintenance.  Both  metrics  typify  most  program  performance 
evaluations  and  ignore  any  consideration  of  the  quality  of 
software  management  [Ref.  1]  .  These  evaluations  assume  all 
software  management  is  the  same  and  doesn't  detract  from,  or 
add  to,  conclusions  derived  from  the  other  metrics.  In 
1981,  Barry  Boehm  [Ref.  1]  wrote. 

Poor  management  can  increase  software  costs  more  rapidly  than  any  other 

factor. 

On  this  basis,  software  management  can  be  considered 
the  third  leg  of  what  is  referred  to  as  the  golden  triangle 
of  software  metrics. 

B.  SOFTWARE  MANAGEMENT  COMPONENTS  AND  GOALS 

Software  management  quality  comes  in  a  wide  variety  of 
forms,  but  most  deal  with  four  distinct  areas:  requirements. 
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estimation/planning,  personnel  and  risk  management.  Current 
papers  on  software  management  are  very  subjective. 
Anecdotal  stories  are  detailed,  case  studies  are  outlined, 
and  bits  and  pieces  of  good  advice  are  presented  [Ref.  2]  . 
Emphasis  is  placed  in  pointing  out  problems  instead  of 
solutions,  and  very  little  in  providing  objective  indicators 
to  measure  management  quality.  This  thesis  builds  an 
objective,  repeatable  metric  to  determine  quality 
management,  measure  improvement,  and  predict  future  success 
levels  of  projects. 

The  goal  is  to  determine  a  structured  set  of  inquiries 
to  quantitatively  measure  software  management  quality.  The 
inquiries  are  organized  into  a  questionnaire  and  minimize 
open-ended  subjective  essay- type  answers.  The  questions  are 
designed  to  confine  responses,  with  the  answer  to  be 
correlated  to  a  standardized  measure.  Three  software 
programs  will  be  examined  for  establishment  of  these 
criteria.  The  three  software  programs  are  the  Suirveillance 
Towed  Array  Sensor  System  ■  (SURTASS)  1989,  the  Financial 
Information  Support  System/ Expenditure  Tracking  System 
(FISS/ETS)  1998,  and  the  Tactical  Environmental  Support 
System/  Naval  Integrated  Tactical  Environmental  System 
(TESS/NITES)  1999.  These  programs  are  cross  sections  of 
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typical  Department  of  Defense  software  development  and 
maintenance  efforts,  and  serve  to  illustrate  varied  software 
management  practices.  In  order  to  encourage  complete 
openness  with  the  survey,  the  results  of  the  surveys  will 
not  identify  which  of  the  three  programs  they  refer  to. 
Instead  the  three  programs  will  be  randomly  referred  to  as 
program  A,  program  B,  and  program  C. 

Collectively,  measures  in  the  following  four  areas  will 
give  an  objective  view  on  the  quality  of  the  software 
management.  Thus,  two  programs  scoring  equally  on  product 
and  process  metrics  can  be  further  measured  and  compared  on 
the  basis  of  the  quality  of  their  management.  This  provides 
a  more  comprehensive  look  at  a  software  program. 

1.  Requirements  Management 

Requirements  management  focuses  on  managing  the  process 
of  extracting,  developing,  defining,  and  refining  the 
requirements  of  a  software  program  [Ref.  3].  It  is  not  the 
intent  of  this  thesis  to  develop  a  product  or  process  metric 
for  requirements.  Multitudes  of  product  and  process  metrics 
exist  in  this  area  [Ref.  4]  .  Alan  M.  Davis  and  Dean  A. 
Leffingwell  [Ref.  5]  state  that. 
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Requirements  are  capabilities  and  objectives  to  which  software  must 
conform  and  are  the  common  thread  for  all  development  (and 
maintenance)  activities.  Requirements  management  is  the  process  of 
eliciting,  documenting,  organizing,  and  tracking  changing  requirements 
and  communicating  this  information  across  the  project  team. 
Implementing  (quality)  requirements  management  ensures  that  iterative 
and  unanticipated  changes  are  maintained  throughout  the  project  lifecycle. 

Quality  management  of  a  program's  requirements  must 
establish  procedures  and  structure  to  ensure  that 
requirements  specifications  are  complete,  consistent, 
readable,  lack  ambiguity,  can  be  traced  to  origins,  and  do 
not  arbitrarily  contain  design  stipulation  [Ref.  5]  .  Each 
requirement  should  be  a  singular  idea  [Ref.  3]  .  Good 

management  addresses  the  requirement  attributes.  These 
include  managing  customer  benefit,  the  requirements  author 
and/or  responsible  parties,  the  corresponding  effort,  the 
development  priority,  rationale,  and  relationships  to  other 
requirements.  The  effort  in  tracking  status,  dates,  and 
versions  also  is  a  determinate  of  quality  management  .[Ref. 
5]  . 

2 .  Estiination/Planning  Management 

Estimations  are  the  basis  of  which  planning  is 
performed  on  a  program  [Ref.  6]  .  The  estimation/planning 
management  section  does  not  seek  to  choose  or  purport  a 
specific  estimation  technique.  This  area  seeks  to  quantify 
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the  management  effort  of  the  estimation  process.  The 
questions  are  if  the  choice  of  a  technique  is  appropriate 
and  how  well  that  technique  is  implemented. 

3 .  People  Management 

The  people  management  section  encompasses  not  only  such 
issues  as  the  program  manager's  ability  to  allocate  human 
resources  appropriately  and  ensure  an  appropriate  working 
environment,  it  also  includes  communication  and  leadership. 
This  includes  not  only  the  communication  and  leadership 
skills  of  the  program  manager,  but  also  the  stiructure  set  up 
for  communication  and  mentoring  leadership  for  the  entire 
program.  This  thesis  looks  at  management  of  people  from  a 
specific  software  development /management  perspective.  It 
examines  such  questions  as,  does  management  create  the 
proper  environment  through  good  working  conditions  and  an 
appropriate  reward  structure,  and  does  management  create 
unnecessary  overlaps  or  underlap's  through  poor  organization, 
delegation,  and  task  monitoring.  This  section  is  an 
exclusive  focus  on  the  unique  qualities  and  needs  of  people 
working  in  a  software  development  environment . 

4 .  Risk  Management 

An  overarching  theme  that  runs  through  each  of  these 
sections  is  risk  management.  Ultimately,  it  is  management's 
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ability  to  identify  and  resolve  high-risk  elements  early 
that  will  have  the  greatest  impact  on  the  success  or  failure 
of  a  software  program  [Ref.  7] .  It  is  difficult  to 
objectively  measure  subjective  decisions  regarding  risk 
management.  It  then  elevates  the  priority  to  objectively 
measure  the  effort  and  structure  a  program  has  dedicated  to 
risk  assessments. 
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II .  REQUIREMENTS  MANAGEMENT 


A.  COMPONENTS  AND  CRITERIA 

In  developing  software,  requirements  are  the  reason  why 
it  is  done  in  the  first  place.  Without  requirements,  there 
is  no  need  for  development.  [Ref.  8] 

A  software  development  project  generally  consists  of 
initial  requirements,  refined  requirements,  implementation 
of  requirements,  and  then  testing  of  the  product  for 
conformance  to  the  requirements  [Ref.  9].  A  software 
maintenance  or  follow-on  upgrade  development  deals  with  new 
and  modified  requirements.  A  well -documented  requirement 
is  a  single  idea  or  function  [Ref.  3]  .  The  requirement  is 
easy  to  understand  and  is  testable  in  some  fashion  [Ref.  3]. 
For  these  reasons,  the  management  of  requirements  is  an 
important  measure  of  the  quality  of  program  management.  For 
instance,  can  the  program  manager  control  thd  process  of 
development,  prioritization,  and  implementation  of 
requirements,  given  constraints  in  any  of  these  areas? 
Constraints  can  be  in  the  form  of  mandates  to  employ  a 
certain  development  process,  a  selected  architecture,  or  by 
a  predetermined  set  of  requirements. 
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The  program  manager  must  identify  and  ensure  that  all 
stakeholders  are  involved  in  the  initial  requirements  list 
development.  Failure  to  include  all  parties  at  the  start 
will  most  likely  spell  trouble  down  the  line  [Ref.  10]  . 
Steve  McConnell  [Ref.  11]  ,  in  his  IEEE  Software  article 
listing  Software's  Ten  Essentials,  calls  the  product 
specification,  the  software  program's  compass.  He  states, 

. .  .without  one,  you  can  perform  the  work  of  Hercules  and  still  not  produce 
a  working  product  because  the  work  in  aggregation  hasn’t  been  aimed  in 
any  particular  direction.  Without  good  direction,  any  individual’s  work 
can  go  the  wrong  direction  and  different  people  can  work  at  cross¬ 
purposes. 

Most  program  managers  regard  requirements  as  the 
contract  between  the  developer  and  the  customer  on  a  program 
[Ref.  12].  The  program  manager  manages  customer's 

expectations  by  managing  the  requirements  [Ref.  12]  . 
Generally,  a  program  is  created  to  fill  a  user  or  customer 
need.  In  the  Department  of  the  Navy,  that  could  mean  the 
fleet  has  a  need  for  some  capability.  That  capability  may 
be  translated  into  a  submarine  tracking  software  module.  Or 
in  the  commercial  world,  a  company's  marketing  group  may 
determine  that  a  large  market  exists  for  payroll  tracking 
software.  In  either  case,  when  the  need  for  some  capability 
is  uncovered,  the  end  users  normally  do  not  have  software 
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experience  or  background,  but  do  know  what  the  desired 
results  are.  In  these  cases,  the  end  product  is  manipulated 
via  an  Operator-Machine  Interface  (OMI) . 

The  goal  of  program  management  is  to  convert 
user/customer  needs  into  an  unambiguous  set  of  requirements 
for  the  development  team  [Ref  8]  . 

A  quality  program  manager  will  facilitate  the 
user/customer  needs  into  requirements  that  can  be  coded. 
This  process  happens  in  one  of  two  ways.  The  first  is  the 
direct  procedure .  Users  convey  in  any  number  of  ways  their 
needs  to  program  management,  which  in  turn  develops  the 
formal  requirements  to  which  the  developers  code.  The 
second  is  the  indirect  procedure.  The  users  convey  their 
needs  directly  to  the  programmers  who  rapidly  develop  a 
prototype,  which  the  users  can  see  and  validate.  This 
process  can  be  iterative.  Program  management  adjudicates 
between  user  and  developer  during  the  indirect  process  and 
develops  formal  requirements.  However,  the  formal 
requirements  serve  mainly  as  a  record  of  what  has  been 
performed  [Ref.  13]  . 

Figures  1  and  2  illustrate  program  management's  role  in 
both  approaches  to  requirement  extraction  [Ref.  13] . 


9 


Figure  1 .  Determining  requirements  via  direct  program 

management  involvement 


Figure  2 .  Determining  requirements  via  indirect  program 

management  involvement 
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Many  requirement  definition  techniques  are  available  to 
aid  the  program  manager.  Use-case  diagrams.  Class 

Responsibility  Collaborator  (CRC)  models,  or  other  scenario 
type  documentation  is  used  to  extract  precise  requirements. 
Because  of  past  program  failures  due  to  poorly  planned  or 
derived  requirements,  consensus  is  that  a  program  manager 
must  enact  some  sort  of  formal  process  for  the  extraction 
and  formulation  of  requirements.  [Ref.  8] 

CAPT  Gerry  Nifontoff  (USN  ret.)  [Ref.  13]  states 

...because  of  today’s  tools,  any  software  program  involving  OMI  output, 
must  involve  direct  dialog  between  the  users  and  the  developers. 

The  users  express  to  developers  what  they  need  and.  the 
developers  develop  a  quick  prototype  to  feedback  to  the 
users.  The  process  continues  as  program  management 
facilitates  and  adjudicates  the  process. 

Figure  3  shows  possible  actions  a  program  manager  can 
use  to  define  requirements.  It  is  Scott  Ambler's  [Ref.  8] 
"starburst"  diagram  for  defining  and  validating  initial 
requirements.  This  is  an  iterative  process  moving  from 
center  to  one  of  the  techniques  and  back  again. 
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Figure  3 .  Starburst  Diagram 

This  process,  also  called  Rapid  Application  Development 
(RAD),  is  very  popular  today  [Ref.  14].  Barry  Boehm  [Ref. 
14]  says  that  in  general,  RAD  gives  earlier  product  payback 
and  more  payback  time  before  the  pace  of  technology  makes 
the  product  obsolete.  For  software  product  sales,  RAD  also 
helps  debut  a  product  earlier  in  a  market  window,  which  lets 
the  product  capture  more  market  share,  revenues,  and 


12 


profits.  To  gain  the  maximum  benefit  from  RAD,  the  program 
manager  must  choose  the  RAD  form  that  best  suits  the 
project . 

Another  closely  related  approach  that  is  growing  in 
popularity  is  throwaway  software.  This  concept  is  simple. 
Upon  startup,  the  developer  may  not  know  much,  but  while 
creating  the  software  does  learn  what  users  really  want  and 
how  to  make  clean  code.  By  the  time  the  project  is 
finished,  the  developer  has  learned  so  much  that  it  would  be 
much  better  if  everything  was  thrown  away  and  started  over. 
[Ref.  15] 

The  program  manager's  task  is  to  analyze  a  project  to 
find  the  hardest  parts,  then  implement  the  throwaway 
software  plan  in  these  areas.  [Ref.  15] 

Synchronize  and  stabilize  is  an  approach  that  companies 
such  as  Microsoft  use  to  compete  in  the  fast  paced  markets, 
such  as  Internet  software.  This  model  starts  with  a  vision 
of  what  the  product  should  do .  The  program  manager  derives 
a  rough  functional  specification,  which  the  team  evolves 
until  the  end  of  the  project.  The  schedule  has  multiple 
stabilization  point,  or  milestones.  Three  is  a  common 
number.  Each  stabilization  point  represents  progress  after 
weeks  of  a  development  sub-cycle  and  usually  represents  an 
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alpha  or  beta  release.  Requirements  are  finished  when  the 
development  is  finished  and  the  product  has  been  released. 
[Ref.  16] 

The  requirements  list  alone  is  not  sufficient.  It  is 
the  responsibility  of  the  program  manager  to  establish 
requirements  prioritization  [Ref.  17]  .  Time  and  money 
limitations  apply,  and  decisions  must  be  made  on  which 
requirements  take  precedence  over  the  others .  The  program 
manager  must  ensure  that  a  thorough  assessment  of  all 
tradeoffs  has  been  made.  Outside  factors  play  an  important 
role  in  determining  the  options  a  program  manager  has  in 
this  area.  On  one  side  of  the  spectrum,  a  program  that  is 
limitless  in  funding  and  time  can  afford  the  program  manager 
the  maximum  array  of  options.  In  reality,  restrictions  on 
both  time  and  money  to  complete  a  development. 

Identifying  all  the  requirements  upfront  and  then 
developing  the  product  is  idealistic  in  today's  software 
environment .  Requirements  change  for  many  reasons  [Ref . 
18].  It  is  the  program  manager's  responsibility  to 
establish  some  type  of  change  management.  Change  management 
will  help  you  direct  and  coordinate  those  changes  so  they 
can  enhance  -  not  hinder  -  your  software  [Ref.  18]  .  The 
change  management  procedures  must  be  easy  to  understand  and 
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consistent . 


That  is  not  to  say  that  the  development  is 


subject  to  requirement  changes  at  any  time  during  the 
development.  It  is  well  documented  that  time  and  cost 
increase  exponentially  when  requirements  are  changed  late  in 
the  development  process  [Ref.  4]  .  The  program  manager  must 
choose  to  "freeze"  requirements  at  some  point,  but  establish 
the  framework  for  a  follow-on  version  release  or  block 
upgrade.  The  lesson  learned  from  the  past  ten  years  has 
been  that  software  products  are  unlike  most  durable  goods  in 
that  they  are  always  changing.  For  instance,  when  buying  or 
learning  to  use  a  new  program  or  word  processor,  the  user 
touted  the  view  that  the  system  would  be  long-lived.  The 
user  now  desires  and  expects  updates  or  new  programs  with 
added  features  and  capabilities  fielded  in  less  than  one 
year,  with  the  system  having  a  relatively  short,  useful 
life. 

B .  QUESTIONS 

The  questionnaire  for  requirements  management  evaluates 
the  program  manager  on  establishment  of  procedures.  The 
goal  is  to  tailor  the  software  development  process  (and  its 
management)  to  achieve  optimal  results,  satisfying 
user/customer  wants  and  needs  with  minimal  time  and  money 
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expended.  These  questions  do  not  seek  to  determine  the 
quality  of  judgements  on  any  specific  decisions  made.  The 
thrust  of  the  questions  is  to  establish  the  structure,  if 
any,  laid  out  by  the  program  manager  in  the  area  of 
requirements . 

Each  suirvey  is  designed  to  pertain  to  a  single  program. 
The  pair  choice  and  yes -no  questions  address  three 
encompassing  areas  of  requirements  management .  The  top 
three  areas  are  not  clear-cut  and  may  overlap.  They  are 
extraction,  change,  and  testability. 

1.  Extraction 

Extraction  covers  the  broad  area  of  who  is  involved  in 
the  process,  what  the  processes  are,  and  when  it  is  done. 
Customer  dissatisfaction  and  cost  overruns  are  often  caused 
by  poor  requirements  that  are  produced  by  people  who  do  not 
understand  the  requirements  process  (or  choose  not  to 
implement  one)  [Ref.  3]. 

2 .  Change 

All  programs  have  requirements  change,  with  the  sole 
exception  being  pure  standalone,  throwaway  software.  These 
questions  ascertain  how  change  is  handled.  Are  there  any 
procedures  and  what  is  the  potential  for  creating  stable 
changes  for  the  system? 


3 .  Testability 

What  is  the  program  manager's  view  of  testing 
requirements,  and  where  is  the  emphasis  placed  on  testing? 
Does  the  program  manager  consider  testing  up  front  or 
towards  the  end?  Each  requirement  should  be  testable  [Ref. 
19]  . 

Additionally,  these  questionnaires  deal  with  formality. 
Formality  determines  how  precisely  requirements  'are 
explored,  extracted,  and  recorded.  Is  the  change  process 
well  defined?  And  has  the  test  process  been  thoroughly 
defined?  Whatever  processes  are  used  in  the  program,  they 
must  be  well  understood,  recorded,  and  retrievable. 

Figure  4  graphically  illustrates  the  hierarchy  of 
factors  for  requirements  management . 
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Figure  4 .  Requirements  Management  Hierarchy  Factors 

The  three  areas  directly  under  requirements  management 
indicate  the  next  lower  tier  of  factors  to  evaluate .  The 
dotted  lines  between  requirement  testing,  requirement 
extractions,  and  change  management  indicate  the  iterative 
relationship  between  the  areas  as  a  program  progresses. 
Below  requirement  extractions  are  the  activities  of 
user/stakeholder  identification  and  requirement  refinement 
necessary  for  successful  and  thorough  completion  of 
requirement  extractions. 
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III.  ESTIMATION/PLANNING  MANAGEMENT 


A.  COMPONENTS  AND  CRITERIA 

Planning  is  the  key  to  control.  (Rick  Weber  [Ref.  20],  Time  Management 

Essentials) 

When  one  thinks  about  management,  one  thinks  about 
planning.  Managers  plan  strategy,  schedule,  costs,  etc. 
Software  development  programs  and  planning  have  been  an 
oxymoron  throughout  the  1960s,  1970s,  and  early  1980s. 

Among  software  systems  delivered,  many  were  subject  to  cost 
overruns,  late  delivery,  lack  of  reliability,  inefficiency, 
and  lack  of  user  acceptance.  [Ref.  21] 

The  basis  of  planning  is  estimation.  Planning  a 
software  product  development  requires  a  frame  of  reference 
and  an  ability  to  measure  against  the  reference.  The 
program  manager  has  three  major  measures  to  estimate  the 
program  by  products,  processes,  and  resources  [Ref.  22]  . 
Humphrey  [Ref.  4]  states. 

You  measure  to  get  data,  and  you  want  data  to  help  you  with  the 

following: 

•  Gain  qualitative  understanding 

•  Evaluate  a  product,  process,  or  organization 
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•  Control  a  product  or  a  process 

•  Make  an  estimate  or  a  plan 

Product  measures  generally  refer  to  volume.  Examples 
include  lines  of  code  (LOG) ,  pages  of  documentation,  number 
of  screens,  and  number  of  files.  The  measure  can  be  the 
whole  product  or  various  product  elements,  such  as  modules, 
components,  or  manuals.  Measurement  is  accomplished  by 
phase,  such  as  the  amount  of  code  produced  in  the 
implementation  phase  or  the  LOG  changed  during  unit  testing. 
Measures  of  other  product  attributes  might  include  system 
throughput,  memory  capacity,  cyclomatic  complexity,  module 
coupling,  and  function  points  (FP) .  [Ref.  4] 

Process  measures  quantify  behavior,  strategies,  and 
execution  of  the  process  used  to  develop  the  product .  One 
general  category  of  process  measures  is  event  counts. 
Examples  include  the  number  of  defects  found  in  test, 
requirement  changes,  or  milestones  met.  Another  general 
category  concerns  time  measures.  The  time  required  to 
complete  a  project  is  often  called  cycle  time.  In  highly 
competitive  markets,  cycle  time,  or  deployment,  may  be  more 
important  than  reducing  development  costs  [Ref.  4] .  All  the 
stakeholders  and  the  organization  must  be  considered  and 
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included  in  the  analysis,  planning,  and  implementation 
needed  to  release  software. 

Resource  measures  refer  specifically  to  labor  hours 
required  for  product  development  [Ref.  4].  Boehm  [Ref.  1] 
further  extends  the  measurement  to  include  factors  such  as 
proper  number  and  assignment  of  people  to  the  work,  the 
proper  working  conditions  and  reward  structure  for  people, 
the  proper  resources  (terminals,  support  software  tools, 
etc.),  and  other  quality  management  practices  associated 
with  requirements  management  and  risk  management.  Pressman 
[Ref.  22]  includes  money  as  a  resource  measure.  But  money, 
unless  it  is  a  pre-set,  fixed  and  known  resource,  cannot  be 
properly  included.  Cost  (money)  typically  becomes  an 
estimated  outcome  from  process,  product,  and  resource 
measures . 

Estimation  utilizing  all  three  measures  for  a  program 
will  lead  to  realistic  planning  of  schedules  and  costs. 
Subsequent  tracking  of  metrics  throughout  the  program  will 
aid  program  updates  and  provide  a  solid  basis  to  which 
future  programs  can  planned  against  [Ref.  1] . 

A  simple  example  of  using  estimations  to  provide  an 
initial  program  plan  is  the  LOG  a  programmer  can  code  per 
day.  Estimate  the  product  size  and  number  of  programmers. 
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and  duration  estimates  can  be  determined. 


Include  the 


salaries  of  the  workers  over  the  duration,  and  cost  is 
determined.  With  duration  and  cost  estimated,  an  initial 
program  schedule  can  be  formulated. 

Brooks  [Ref  23]  concedes  that  cost  does  indeed  vary  as 
the  product  of  the  number  of  men  and  the  number  of  months. 


but  emphasizes 

that  progress  does  not . 

Reasons 

include 

the 

inability  to 

adequately 

partition 

tasks 

because 

of 

sequential  constraints 

and  poorly 

drawn 

lines 

of 

responsibility 

due  to 

management  mi s  j  udgment . 

Poor 

correlation  of  consistent  actual  results  also  stems  from  the 
difficulty  in  estimating  the  productivity  of  programmers 
[Ref.  22] .  It  is  estimated  that  differences  in  productivity 
among  the  best  and  worst  programmers  are  commonly  10  to  1 
and  may  be  as  high  as  25  to  1  [Ref.  19] . 

Even  when  tasks  can  be  partitioned,  the  burden  of 
communication  must  be  added  to  the  amount  of  work  to  be 
done,  particularly  the  effort  required  for 
intercommunication.  If  each  part  of  the  task  must  be 
separately  coordinated  with  each  other  part,  the  effort 
increases  as  n  *  (n-l)/2,  where  n  is  the  number  of  people 
needing  to  communicate.  [Ref.  23] 
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Couple  the  productivity  variances  with  other  factors 
such  as  work  environment,  organizational  structure, 
reward/recognition,  training,  and  motivation,  and  the 
importance  of  management  quality  becomes  very  apparent. 

For  large,  complex  software  programs,  a  Work  Breakdown 
Structure  (WBS)  is  recommended  [Ref.  12]  .  A  WBS  defines  all 
important  tasks,  milestones,  and  deliverables  throughout  the 
program  [Ref.  22] . 

Once  initial  costs  and  schedules  are  derived  from 
estimations,  progress  tracking  and  schedules  and  costs 
adjusting  become  key  factors  in  the  software  program  success 
[Ref.  12] . 

Establishing  and  tracking  earned  value  is  recommended 
to  measure  program  progress  [Ref.  12] .  By  assigning  value 
to  a  developer's  work  package,  the  cumulative  value  of 
completed  work  packages  can  be  compared  to  the  estimated  and 
actual  cost  to  complete  the  work  packages  to  give  a  more 
accurate  measure  of  schedule  and  cost  progress  [Ref.  19]. 
Adjusting  schedules  and  costs  later  in  the  program  may 
appear  to  be  an  admission  of  failure  of  the  initial  planning 
effort.  But  Launi  [Ref.  6]  states  that  a  universal  truth 
applies  to  any  project:  change  will  occur  constantly, 
dynamically,  and  usually,  without  warning.  No  matter  how 
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stable  the  initial  estimates  and  plans  seem  to  be,  change 
occur  as  the  program  progresses  for  many  reasons  including 
discovery  of  unknowns  associated  with  the  product,  process, 
or  resources.  It  is  a  measure  of  software  management 
quality  as  to  how  the  changes  are  handled  [Ref.  12] . 

The  program  manager  must  set  up  a  structure  to  use 
product,  process,  and  resource  measures  in  a  software 
program,  and  it  is  the  program  manager's  responsibility  to 
ensure  that  the  measure  being  used  will  yield  the  most 
accurate  and  useful  results  possible  for  the  software 
program . 

B .  QUESTIONS 

The  questions  in  this  section  ascertain  that  the 
program  manager  is  performing  both  initial  and  follow  up 
estimation  and  planning.  The  questionnaire  checks  that 
derived  documentation  is  completed  and  used  in  the  program. 
Moreover,  it  is  used  to  determine  if  currently  accepted 
methods  and  practices  are  being  employed.  Is  the  program 
manager  managing  the  estimation  and  planning  process 
sufficiently  to  give  confidence  to  the  product,  process,  and 
resource  measures  gathered?  No  attempts  were  made  during 
interviews  with  program  managers  or  discussions  with  focus 


24 


groups  to  determine  which  measure  or  method  is  superior. 
The  questions  are  designed  to  solicit  the  best  structure  and 
its  practices. 

Figure  5  graphically  illustrates  the  main  hierarchy 
factors  in  estimation/planning  management. 
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Figure  5.  Estimation/Planning  Management  Hierarchy  Factors 
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IV.  PEOPLE  MANAGEMENT 


A.  COMPONENTS  AND  CRITERIA 

The  people  management  section  of  the  thesis  evaluates 
the  software  program  manager  in  two  ways:  the  skills  that 
the  software  program  manager  exhibits  and  the  type  of 
organizational  structure  instituted  by  the  program  manager. 

If  a  single  person  could  perform  all  of  the  programming 
and  software  work  on  a  product,  there  would  be  no  need  for 
people  management.  How  management  recruits,  organizes,  and 
treats  human  resources  is  instrumental  to  the  success  or 
failure  of  any  endeavor  involving  many  persons  [Ref.  22]  . 
Software  development  requires  highly  skilled  professionals. 
Unlike  producing  widgets,  software  is  a  product  of  the  mind. 
Although  automated  tools  aid  the  developer,  software  is 
still  largely  based  on  individual  interpretation  and 
implementation. 

1 .  Human  Resources 

The  program  manager  must  recruit,  train,  allocate  tasks 
and  teams,  and  reinforce  positive  behaviors  for  an  overall 
working  environment  that  increases  a  program's  chance  for 
success  [Ref.  12]  .  Techniques  that  foster  such  an 
atmosphere  includes  showing  appreciation,  injecting  humor 
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whenever  possible  and  empowering  team  members  [Ref.  24]  . 
In  some  organizations,  particularly  those  like  the 
Department  of  Defense,  factors  may  limit  the  ability  of  the 
program  manager  to  recruit,  select,  or  otherwise  change  the 
software  development  team  members.  Restrictive 
organizations  necessitate  the  program  manager  maximize 
existing  human  resources  by  concentrating  on  activities  such 
as  training  and  reinforcing  positive  behaviors  to  create  a 
successful  program  environment  [Ref.  13]  . 

Training  is  often  seen  as  a  frill  in  many 
organizations,  something  to  be  reduced  in  order  to  meet 
profit  goals  in  times  of  economic  stringency.  However, 
training  can  be  a  source  of  competitive  advantages  and  is  an 
integral  component  to  an  overall  productive  management 
practice  [Ref.  25]  .  Software  development  programs  with 
tight,  hectic  schedules  are  not  an  excuse  for  elimination  or 
necessarily  a  good  reason  for  postponement  of  training  [Ref. 
12]  .  The  program  manager  must  carefully  plan  training  into 
the  framework  of  the  overall  program  schedule  to  ensure  the 
organization  of  its  long-term  benefits  without  endangering 
short-term  program  needs  [Ref.  25]  . 

Luthans  and  Stajkovic  [Ref.  26]  state. 
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The  real  challenge  (of  software  program  management)  is  to  find  ways  to 
manage  human  resources  as  effectively  as  possible  in  order  to  attain 
world-class  performance. 

Reinforcing  for  performance  is  a  tool  the  program 
manager  can  utilize  to  promote  positive  behaviors  and 
eliminate  negative  behaviors.  Organizational  Behavior 
Modification  (0.  B.  Mod)  is  a  systematic  approach  based  on 
reinforcement  theory.  Reinforcement  theory's  basic  premise 
is  that  human  behavior  is  a  function  of  contingent 
consequences.  Something  that  strengthens  and  leads  to  an 
increase  in  the  frequency  of  a  behavior  is  called  a 
reinforcer,  not  a  reward.  Software  program  managers  may  not 
get  desired  behavior  from  individuals  with  pay  and  rewards 
alone.  By  reinforcing  using  O.  B.  Mod  procedures,  one 
always  increases  the  strength  and  frequency  of  the  desired 
functional,  performance-related  behaviors.  Therefore, 

performance  improvements  will  always  result  from  reinforcing 
for  performance.  [Ref.  26] 

0.  B.  Mod  consists  of  five  steps:  identify,  measure, 
analyze,  intervene,  and  evaluaite.  The  approach  seeks  to 
identify  the  critical  obseirvable  performance-related 
behaviors,  measure  the  baseline  frequencies  of  the  critical 
behaviors,  analyze  to  determine  the  antecedents  and 
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contingent  consequences  in  the  performance -related  context, 
intervene  to  increase  the  frequency  of  the  positive 
behaviors  and  decelerate  the  dysfunctional  behaviors,  and 
finally,  evaluate  for  performance  improvement.  [Ref.  26] 


Figure  6.  Adaptation  of  0.  B.  Mod  application  model 

The  use  of  the  0.  B.  Mod  application  model  is 
demonstrated  in  a  service-sector  example.  Bank  supervisors 
used  contingently  administered  feedback  and  social 
recognition  and  attention  reinforcers  for  teller  customer 
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service  behaviors.  This  included  using  the  customer's  name, 
providing  a  balance,  and  making  eye  contact.  These 
behaviors  led  to  increases  in  measured  customer 
satisfaction.  In  this  same  bank,  the  earlier  use  of 
monetary  rewards  had  had  no  measurable  effect  on  customer 
satisfaction.  The  money  turned  out  to  be  a  reward;  not  a 
contigently  administered  reinforcer  that  strengthened  teller 
customer  service  satisfaction.  [Ref.  26] 

A  corollary  example  specific  for  software  development 
could  focus  on  reduction  of  individual  programming  errors,  a 
primary  factor  in  determining  software  product  quality  [Ref. 
27] .  By  identifying  and  measuring  the  critical  behaviors 
that  programmers  demonstrated  when  writing  good,  error- free 
code,  program  management  can  then  analyze  the  behavior 
contingencies  and  develop  an  intervention  strategy.  The 
program  manager  can  then  use  one  or  more  of  the  three  types 
of  reinforcers ;  money,  feedback,'  and  social;  to  promote  the 
behavior  leading  to  fewer  errors  in  delivered  code.  The 
program  manager  can  evaluate  this  improvement  in  performance 
against  measures  of  costs  and  schedule.  Reduced  program 
costs  and  meeting  schedule  dates  are  direct  results  from 
reducing  programming  errors  [Ref.  28]  .  Therefore,  it  is 
concluded  that  the  use  of  reinforcers  can  help  the  software 
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program  manager  effectively  manage  human  resources  to 
achieve  desired  behaviors  and  results  from  the  software 
de ve 1 opment  team. 

To  date,  improvement  programs  for  software 
organizations  have  often  emphasized  process  or  technology, 
not  people  [Ref.  29] .  The  Software  Engineering  Institute's 
(SEI)  People-Capability  Maturity  Model  (P-CMM)  was  patterned 
directly  after  the  SEI  CMM  structure.  While  the  CMM  focuses 
on  software  processes  and  practices,  the  P-CMM  concentrates 
on  a  software  organization's  human  resource  management  and 
development.  .  The  purpose  of  the  P-CMM  is  to  improve  a 
software  organization's  ability  to  attract,  develop, 
motivate,  organize,  and  retain  talent  needed  to  steadily 
improve  software  development  capability,  via  encouragement 
to  meet  high  activity  level  standards.  [Ref.  29] 

As  with  the  CMM,  the  level  one  for  P-CMM  is  the  initial 
level,  the  ad  hoc  activity  level.  Level  2  seeks  to  instill 
basic  discipline  into  workforce  activities.  In  level  3, 
management  identifies  primary  competencies  and  aligns 
workforce  activities  with  them.  Level  4  has  quantitatively 
managed  organizational  growth.  Competency-based  teams  and 
practices  are  used. 
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improving  individual  competencies  and  innovative  ways  to 
improve  workforce  motivation  and  capability.  Coaching,  more 
formalized  and  greater  depth  assistance  is  provided  to  both 
individuals  and  teams.  The  organizational  culture  is 
created  and  evolves,  as  all  members  of  the  workforce  are 
striving  to  improve  the  individual,  team,  and  unit 
knowledge,  skills,  and  motivation.  [Ref.  29] 

P-CMM  is  concerned  with  the  issues  that  primarily  come 
under  the  human  resources  section  of  people  management. 
Over  levels  two  through  five,  twenty  key  process  areas  are 
described.  However,  those  twenty  key  process  areas  roughly 
cover  four  general  areas:  individual  motivation,  individual 
development,  team  forming,  and  team  development.  The  result 
is  an  organizational  culture.  An  organization's  culture  is 
manifest  when  its  members  share  core  values  that  guide  their 
behavior  [Ref.  30].  An  organization  that  lacks  repeatable 
management  or  development  practices  does  not  have  a  culture 
[Ref.  30] . 

2 .  Leadership 

Software  managers  have  the  crucial  role  in  establishing 
culture.  Leadership  from  software  managers  comes  before 
process  or  organization  and  the  capability  model  makes  no 
overriding  differentiation  [Ref.  31] . 


Therefore,  software  program  managers  are  responsible 
for  providing  the  leadership  to  enable  good  practices  for 
managing  human  resources.  While  there  are  many  different 
styles  and  personalities  involved  in  management,  each  with 
its  own  strengths  and  weaknesses,  a  cross  section  of 
positive  behaviors  have  been  identified  [Ref.  32]  .  These 
behaviors  are  based  on  the  Myers -Briggs  Type  Indicators 
(MBTI)  .  MBTI  was  developed  from  the  psychological  type 
theory  work  of  Carl  Jung  [Ref.  33] . 

The  four  scales,  each  with  two  opposite  poles,  broadly 
covers  all  areas  that  a  manager  would  be  characterized.  The 
four  areas  are:  attention  focus  (Extrovert  vs.  Introvert); 
information  gathering  (Sensing  vs.  Intuition);  decision¬ 
making  (Thinking  vs.  Feeling);  and  orientation  towards  the 
outer  world  (Judging  vs.  Perceiving).  Based  on 


combinations,  there 

are 

sixteen  distinct 

patterns 

of 

behaviors 

defined. 

The 

MBTI  survey  is 

devised  as 

a 

repeatable 

obj  ective 

view 

on  where  the  tendencies  of 

a 

person  lay.  A  series  of  questions  is  presented  with  a 
choice  between  two  words  or  phases  that  best  describe  the 
preference.  Based  upon  the  totals,  the  preference  is  mapped 
onto  the  respective  scale  for  each  area.  [Ref.  33] 
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There  is  no  right  or  wrong  judgement  associated  with 
the  MBTI  scale  preferences.  The  preference  identification 
is  meant  as  an  evaluation  of  where  individual  strengths  and 
weaknesses  lie.  Street  [Ref.  34]  believes  that  leaders 
whose  Type  Indicator  preferences  tend  toward  any  of  the 
sixteen  personality  preferences  in  the  MBTI  can  be 
successful.  Each  personality  should  work  to  expand  the 
natural  positive  type  traits  and  minimize  the  negative 
traits,  or  substitute  more  conducive,  unnatural  behaviors 
[Ref.  33]  . 

Based  on  MBTI,  Burgess  and  Street  [Ref.  32]  developed 
the  Wave  Model.  The  Wave  Model  defines  five  areas  that  a 
successful  supervisor  must  excel  in.  These  areas,  in  order, 
are  personal  skills,  interpersonal  skills,  team  skills, 
leadership  skills,  and  organization  skills.  Successfully 
understanding  and  implementing  each  area  successively  builds 
upon  the  next,  that  is,  organizational  skills  can  be 
mastered  only  after  the  prior  four  areas  are  mastered. 
[Ref.  32] 

Personal  job  satisfaction  and  subsequent  productiveness 
relies  more  on  the  micro-work  environment  than  the  macro¬ 
work  environment  [Ref.  13] . 


LIKERT’S  FOUR  LEADERSHIP  PHILOSOPHIES* 


SYSTEM  1  SYSTEM  2  SYSTEMS  SYSTEM  4 


(Exploitative 

Autocratic) 


People  are  seen  as 
basically  lazy,  selfish, 
dishonest,  and  inept; 
they  will  not  work 
unless  constantly 
threatened  and  closely 
supervised;  workers 
are  exploited  and  have 
little  recourse. 


•  People  are  motivated 
by  the  fear  of  the  loss 
of  job,  pay,  or  dignity; 
they  will  be  terminated 
or  punished  if  they  do 
not  comply  with 
management’s 
directions;  “it’s  my 
way  (the  bosses)  or 
the  highway,” 

*  Knowledge^  ability, 
and  creafivity  are  seen 
as  concentrated  in 
management;  workers 
are  seen  as  largely 
incompetent;  as  a 
result,  there  is  no 
need  for  management 
to  consult,  because 
labor  has  nothing 
useful  to  say. 


•  To  best  control  labor, 
work  is  divided  into 
small  (“dumber  and 
dumber”)  pieces; 
there  is  a  supervisor 
for  every  6-8  workers, 
a  manager  for  each  6-8 
supervisors  to  tightly 
control,  direct,  and 
punish;  results  in  a 
steep,  high  hierarchy. 

•  This  is  a 

“master/slave”  style;  it 
is  clear  that  the  worker 
is  not  important  to  the 
organization;  “if  you 
don’t  like  this  deal, 
there’s  a  bus  leaving  5 
minutes;”  its  only 
positive  aspect  is  that 
It  is  honest  about  not 
caring  about  the 
worker;  fear  and 
mistrust  characterize 
relationships. 


(Benevolent 

Autocratic) 


•  Not  much  shift  from  SI ; 
people  are  still  seen  as 
self-centered  and  in 
need  of  close 
supervision;  because 
management  wants  to 
prevent  costly  turnover, 
however,  policies  are 
more  benevolent. 


•  In  addition  to 
fear/punishment,  status 
is  added  as  a 
motivator;  If  workers 
are  mindlessly  loyal 
and  compliant,  they  are 
rewarded  with  the 
illusion  of 
advancement;  S2 
organizations  usually 
have  many  status 
layers  with  each  layer 
having  many  pay 
“steps.” 

•  Knowledge,  ability,  and 
creativity  are  still  seen 
as  concentrated  in 
management;  some 
confidence  is  shown  in 
the  technical  ability  of 
workers;  but 
organizational 
decisions  are  still  made 
without  consultation. 

•  Work  is  still  broken  into 
pieces  with 
management 
responsible  for  the 
Integration  of  work; 
“critical  parent-child” 
relationship  between 
management  and  labor 
(and  between  each 
layer  in  the  steep 
hierarchy). 

•  This  style,  while  more 
benevolent,  is 
manipulative; 

“masters”  treat  the 
“servants”  better 
because  “good  help  is 
hard  to  get,”  but  there 
is  still  no  say  for  the 
servants  on 
“management’  issues; 
mistrust  often 
characterizes  the 
relationships. 


(Consultative)  (Participative) 


•  A  major  shift  from 
S1/S2;  people  are  seen 
as  wanting-even 
needing- to  do  a  good 
job;  if  they  know  what 
needs  doing  and  have 
the  skills,  they  will  do 
a  good  job  without 
very  much  external 
control  or  direction. 

•  Once  the  basic 
“hygiene”  factors 
(pay.  benefits,  working 
conditions,  safety, 
etc.)  are  taken  care  of 
In  a  “fair”  way,  then 
motivation  is  seen  as 
coming  from  within 
the  work;  it  must 
provide  challenge, 
growth,  recognition, 
and  a  sense  of 
contribution. 

•  Knowledge,  ability, 
and  creativity  are  seen 
as  widely  distributed; 
management  does  not 
know  all  the  answers 
(or  even  all  the 
questions):  it  needs 
help  if  the  best 
decisions  for  the 
customer  and  the 
organization  are  to  be 
found;  consultation  is 
the  norm;  less 
hierarchy  is  needed. 

•  Work  is  seen  as 
complex  processes 
involving  networks  or 
employees  working 
together  to  reach 
goals;  management’s 
responsibility  is  to 
create  a  culture 
(values,  strategies, 
structures,  and 
systems)  that  allow  for 
maximum 
consultation. 

•  This  s^le  is  “adult- 
aduir  in  relationship; 
management  is  still 
accountable,  but  it 
recognizes  that  It  must 
consult  widely  if  good 
decisions  are  to  be 
made. 


Very  similar  to  S3; 
people  are  seen  as 
wanting-even 
needing-  to  do  a  good 
job;  if  they  know  what 
needs  doing  and  have 
the  skills,  they  will  do 
a  good  job  without 
very  much  external 
control  or  direction. 


Once  the  basic 
“hygiene”  factors 
(pay.  benefits,  working 
conditions,  safety, 
etc.)  are  taken  care  of 
in  a  “fair”  way,  then 
motivation  is  seen  as 
coming  from  within 
the  work;  it  must 
provide  challenge, 
growth,  recognition, 
and  a  sense  of 
contribution. 


People  are  seen  as 
being  so  capable  that 
many  responsibilities 
seen  in  past  as  being 
solely  the  work  of 
managers  can  be 
transferred  to  self- 
directed  work  teams 
who  perform  these 
leadership 
/management 
functions  as  a  natural 
part  of  getting  the 
technical/tasK  work 
done. 


•  Work  is  seen  as 
complex  processes 
involving  collectives 
of  employees  working 
together  to  reach 
goals;  teams  are 
responsible  for 
task/technical, 
managerial,  and 
leadership  functions. 

•  This  style  is  “adult- 
adulf’  in  relationship; 
management  (and 
team  leaders  with 
delegated 

responsibility)  is  still 
accountable,  but 
recognizes  it  must 
play  a  stewardship 
role  in  creating 
empowered  work 
teams. 


*  Adapted  from  Rensis  Likert,  The  Human  Organization.  (New  York:  McGraw-Hill,  1967) 


Figure  8.  Likert's  Four  Leadership  Philosophies 
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Likert's  Leadership  Philosophies  [Ref.  35]  define  four 
distinct  organizational  working  environments.  Every 
organization  can  be  categorized  as  one  of  these  four  systems 
(or  some  combination  thereof)  .  System  1  and  2  are  closely 
related.  The  basic  premise  of  system  1  and  2  is  that  the 
program  manager  makes  all  decisions,  team  members  are  not 
included  in  decision  making.  The  team  members  may  be  valued 
for  technical  skills,  but  work  is  segmented  into  controlled 
pieces.  The  team  member's  relationship  to  management  is 
more  of  a  master-to- servant .  Systems  3  and  4  are  also 
closely  related.  The  basic  premise  of  systems  3  and  4  is 
that  team  members  are,  to  varying  extents,  part  of  the 
decision  making  process.  Team  member  responsibilities  are 
not  strictly  segmented  and  relationship  to  management  is 
more  of  an  adult- to-adult  type.  [Ref  35] 

Regardless  of  an  overall  organization  system  type, 
every  program  manager  determines  what  system  type  the 
program  will  reflect  (the  micro -work  environment) .  Focus 
group  data  indicates  the  overall  organizational  system 
status  is  a  lesser  factor  on  productivity  of  an  individual 
when  the  program  manager  successfully  implements  system  3  or 
system  4  practices  within  the  program  team.  [Ref.  36] 


38 


3. 


Communication 


Communication  is  the  highest  single  component  in 
measuring  the  quality  of  software  program  management  [Ref. 
12] .  Communication  includes  that  of  the  program  manager 
directly  (vertical)  ,  the  structure  set  up  by  the  program 
manager  for  the  development  team  (horizontal) ,  and  that  with 
the  stakeholders  and  users  (external) . 

Loomis  [Ref.  37]  says. 

Unlike  many  other  industries,  the  software  business  does  not  have  large 
stores  of  tested,  standardized  parts  to  draw  fi'om  in  constructing  new 
systems.  Without  standardization,  communication  of  the  details  becomes 
even  more  essential. 

Whether  directly  involving  program  management  or  others 
associated  with  the  program,  communications  must  be  fostered 
and  promoted  by  program  management  [Ref.  13]  . 

Pickering  [Ref.  35]  describes  and  promotes  the  Network 
Talent  Model  as  an  alternative  to  the  rigid  Industrial 
Model.  The  Industrial  Model  was  developed  at  a  time  when 
the  workforce  generally  had  lower  education  and  performed 
tasks  of  much  less  sophistication.  With  well-defined  and 
limited  skill  roles,  the  common  notion  is  that  technical 
persons  do  not  need  to  perform  management  or  leadership 
skills  and  that  management  persons  do  not  require  technical 
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expertise.  This  is  generally  not  the  case  today  and 
certainly  is  not  the  case  with  software  development 
programs . 

In  contrast,  the  Network  Talent  Model  (NTM)  depicts 
each  individual  in  a  group,  team,  department,  or 
organization  as  possessing  some  necessary  level  of 
management,  leadership,  technical,  and  team  skills.  Work  is 
more  complex  and  individuals  need  to  take  greater  roles  than 
just  their  assigned  tasks.  Everyone  takes  ownership  of  the 
product  or  service  and  participates  in  the  direction  of 
their  company  or  organization's  future.  [Ref.  38]  Present 
day  software  programs  are  highly  complex  and  necessitate 
communication  at  all  levels  [Ref.  12]. 

Hierarchical  structure  exists  in  both  models,  but  the 
NTM  will  vairy  in  the  specific  levels  of  leadership, 
management,  and  technical  responsibilities.  A  top-level 
individual  has  different  leadership,  management,  and 
technical  requirements  and  responsibilities  than  a  lower 
level  individual.  Individuals  participating  in  a  software 
program  are  more  productive  in  a  Network  Talent  Model  than 
an  Industrial  Model  [Ref.  38]  .  Figure  9  illustrates  the 
roles  of  individuals  in  the  respective  models.  [Ref.  35] 
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Figure  9.  Role  types  in  Industrial  Model  (top)  and  Network 

Talent  Model  (bottom) 
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Although  the  program  manager  may  be  hampered  by  the 
overall  organizational  structure  regarding  vertical 
communication  upward,  the  program  manager  is  responsible  for 
ensuring  effective  internal  horizontal  communication  among 
team  members  and  internal  vertical  communication  between 
team  members  and  the  program  management .  To  the  extent 
possible,  the  program  manager  should  also  foster  the 
external  communications  among  users,  stakeholders,  program 
management,  and  development  team  members  [Ref.  12]. 

The  challenge  is  to  encourage  open  lines  of 
communication,  while  residing  within  an  organizational 
structure.  Individuals  vary  in  their  ability  to 
communicate;  actions  taken  by  the  program  manager  will 
either  improve  or  worsen  the  natural  communication 
tendencies  of  individuals  and  teams. 

B .  QUESTIONS 

Because  the  people  management  section  encompasses  many 
distinct  areas  that  are  highly  weighted  in  importance,  the 
questionnaire  is  divided  into  three  sections;  human 
resources,  leadership,  and  communication.  Questions  are 
directed  for  consideration  of  human  resource  management. 
The  leadership  questions  reflect  the  personal  leadership 
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skills  exhibited  and  the  leadership  mentoring  provided  by 
the  program  manager.  The  communication  questions  seek  to 
ascertain  the  communication  protocols  set  up  for  the  program 
organization  and  used  individually  by  the  program  manager. 

Overall,  the  questions  do  not  attempt  to  type  the 
program  manager.  Since  the  people  management  section  is 
paramount  to  determining  management  quality,  these  questions 
seek  to  su2rvey  and  query  for  the  more  conducive  structure 
needed  for  a  successful  software  program  manager.  Figure  10 
illustrates  the  hierarchy  of  factors  in  people  management. 


Figure  10.  People  Management  Hierarchy  Factors 
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V. 


RISK  MANAGEMENT 


A.  COMPONENTS  AND  CRITERIA 

Wiegers  [Ref.  7]  defines  risk  as  a  problem  that  has  not 
happened  yet  but  could  cause  some  loss  or  threaten  the 
success  of  one's  program  if  it  did.  These  potential 
problems  might  have  an  adverse  impact  on  the  cost,  schedule, 
or  technical  success  of  the  program;  the  quality  of 
products;  or  team  morale.  Because  no  program  has  ever  run 
exactly  as  planned,  every  software  program  carries  with  it 
some  degree  of  risk  [Ref.  39].  Therefore,  requirements 
management,  estimation/planning  management,  and  people 
management  all  contain  risks. 

Uncertainty  is  the  unknown  of  what  lies  ahead. 
Attaching  probabilities  to  future  events  changes  uncertainty 
into  risks.  [Ref.  39] 

Risk  management  is  the  process  of  identifying, 
addressing,  and  eliminating  potential  problems  before  they 
can  do  damage  [Ref.  7]  .  It  is  included  as  a  separate 
section  and  separate  factor  in  this  thesis  because  it  is 
critical  in  measuring  the  management  quality  of  a  software 
program  [Ref.  12,  13]  . 
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Figure  11  is  the  SEI  risk  management  paradigm  that 
defines  a  continuous  set  of  activities  that  must  be 
undertaken  to  identify,  communicate,  and  resolve  software 
risks  [Ref.  40] . 


Figure  11.  Risk  Management  Paradigm  [Ref.  40] 

1.  Risk  Assessment 

Risk  assessment  is  the  action  of  examining  a  program 
and  identifying,  areas  of  potential  risk.  Risk  assessment 
encompasses  the  tasks  of  risk  identification,  risk  analysis, 
and  risk  prioritization.  [Ref.  7] 


46 


a.)  Identification 


Identifying  risks  must  be  done  individually. 
Keuffel  [Ref.  39]  classifies  both  macro  and  micro  risks. 
The  macro  risks  are  used  to  measure  the  probability  that 
specified  tasks  will  be  completed  on  specified  dates,  or 
that  specified  functionality  will  be  contained  within  the 
product  under  construction.  It  compares  the  project's 
potential  benefits  with  the  overall  costs  and  risks  required 
to  reach  completion. 

The  micro  view  of  risk  management  is  the  process 
of  breaking  a  project  into  its  component  parts  and 
identifying  each  variable.  Since  this  is  a  painstaking 
process,  Keuffel  [Ref.  39]  suggests  injecting  logic  by 
considering  the  normal  distribution  range  that  each  variable 
may  occupy,  not  the  possible  range.  This  delineation  would 
include  risks  of  the  lead  programmer  leaving  to  work  for  a 
competitor  and  exclude  the  risks  of  the  lead  programmer 
being  struck  by  lightning. 

Although  each  program  is  unique,  the  program 
manager  can  use  history  of  similar  size  programs  to  identify 
risks.  The  use  of  the  Software  Engineering  Institute's 
(SEI)  checklist  of  possible  risk  factors  or  an 
organization's  internal  list  is  another  good  choice  as  the 
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program  manager  considers  checklist-based  evaluations.  [Ref. 
7] 

Although  both  practices  are  utilized,  in  risk 
management,  the  bottom-up  approach  is  viewed  more  favorably 
than  any  top-down  evaluation  [Ref.  39] .  Following  this  line 
of  reasoning,  program  managers  that  hold  team  sessions  and 
get  people  involved  in  developing  the  product  to  participate 
in  risk  management  tend  to  have  a  better  perspective  on  the 
risks  associated  with  the  program. 

b)  Analysis 

Risk  analysis  involves  examining  how  your  program 
outcome  might  change  with  modification  of  risk  input 
variables  [Ref.  7]. 

c )  Priori tization 

Risk  prioritization  helps  to  focus  the  program  on 
its  most  severe  risks  by  assessing  the  risk  exposure. 
Exposure  is  the  product  of  two  factors:  the  probability  of 
incurring  a  loss  due  to  the  risk,  and  the  potential 
magnitude  of  that  loss.  [Ref.  7] 

2 .  Risk  Control 

Risk  control,  although  listed  separately  in  the  SEI 
Risk  Management  Paradigm,  encompasses  risk  planning,  risk 
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tracking,  and  risk  resolution.  Risk  control  is  the  process 
of  managing  risks  to  achieve  desired  outcomes  [Ref.  7]  . 

a)  Planning 

Risk  planning  involves  developing  actions  to 
mitigate  individual  risks,  prioritizing  actions,  and 
integrating  them  into  an  executable  risk  management  plan 
[Ref .  40] . 

b)  Tracking 

Risk  tracking  involves  monitoring  the  status  of 
risks  and  their  mitigation  actions  along  with  the  use  of 
metrics  and  triggering  events  [Ref.  40] . 

c)  Resolution  (Control  in  SET  model) 

Risk  resolution  is  the  execution  of  the  plans  for 
dealing  with  each  risk  [Ref.  7] . 

3 .  Risk  Communication 

Communication  refers  to  the  exchanging  of  risk 
management  information  among  the  functions  and  at  all  levels 
of  the  organization.  This  activity  is  represented  in  the 
center  of  the  model  to  emphasize  its  pervasiveness  and 
criticality  for  implementing  the  other  activities  in  the 
paradigm.  [Ref.  40] 
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4. 


Risk  Avoidance 


Risk  avoidance  is  one  way  the  program  manager  can  deal 
with  a  risk:  do  not  do  the  risky  things!  You  may  avoid 
risks  by  not  undertaking  certain  parts  of  the  program,  or  by 
relying  on  proven  rather  than  cutting-edge  technologies  when 
possible.  [Ref.  7] 

5 .  Regret  Matrix 

The  Regret  Matrix  is  part  of  the  decision  theory  that 
further  quantifies  risk  management  by  attaching 
probabilities  to  future  events.  This  changes  uncertainty 
into  risk,  which  allows  a  calculation  of  net  present 
benefit.  Regret  analysis  performed  on  a  risk  evaluates 
potential  actions  the  manager  may  take  and  its  effect  on  the 
project.  Impact  effect  scales  are  used;  low,  medium,  high, 
in  addition  to  some  absolutes  like  no  effect  and  program 
cancellation,  to  arrive  at  the  best  mitigation  action  to 
follow.  In  general,  using  actual  measurements,  like  a 
function  point  count  of  100,  to  trigger  a  program  risk, 
yields  to  mathematical  modeling  and  is  perceived  as  more 
favorable  than  ordinal  rankings  of  low,  medium,  or  high. 
[Ref.  39] 

The  cost  of  resolving  risk  is  relatively  low  early  on, 
but  increases  as  the  program  progresses  [Ref. 12].  The 
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failure  of  the  program  manager  to  acknowledge  and  implement 
some  level  of  risk  management  is  an  egregious  error  and 
objectively  decreases  management  quality  [Ref.  12].  Thus, 
quality  management  must  include  performance  of  some  type  of 
formal  risk  management.  How  well  a  risk  management  plan  has 
been  implemented  can  be  determined  in  retrospect.  The  risk 
management  factor  of  the  quality  management  metric  can  only 
measure  the  risk  management  structure  set  up.  The  factor 
takes  into  account  any  structure  that  promotes  success  in 
the  software  development  environment  by  considering 
individual  risks,  assessing  individual  impact,  determining  a 
probability  of  occurrence,  and  planning  a  mitigation 
strategy.  Program  management's  judgements  within  the 
established  structures  will  vary,  and  can  ultimately 
determine  the  success  or  failure  of  a  risk  management 
effort.  However,  the  establishment  of  structure  dedicated 
to  these  practices  can  be  objectively  measured  and  yield  a 
strong  indication  of  the  quality  of  program  management. 

An  example  of  the  importance  of  risk  management :  the 
SURTASS  program  had  at  least  two  dramatic  external  changes 
that  changed  the  mission  of  the  development  program.  First, 
in  the  mid-1980s,  the  Toshiba  Corporation  of  Japan,  sold  the 
Soviet  Union  advanced  milling  equipment  that  enabled  the 
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Soviet  Union  to  produce  quieter  submarines ,  The  program 
requirements  changed  significantly  as  the  focus  shifted  from 
passive  to  active  functionality.  Secondly,  in  the  late 
1980s,  the  collapse  of  the  Soviet  Union  dramatically  altered 
the  mission  need  of  the  program  and  impacted  the  planned 
production.  The  goal  of  risk  management  is  to  anticipate 
these  possible  risks  and  have  mitigation  plans  in  place  for 
necessary  alterations  to  the  development  program  [Ref.  13] . 

B .  QUESTIONS 

The  questionnaire  in  this  section  will  ascertain  the 
structures  used  by  program  management  for  identification, 
monitoring,  and  managing  risk.  The  questions  dete2rmine 
whether  the  program  manager  has  set  in  place  strategies  and 
personnel  to  thoroughly  implement  risk  assessment,  explore, 
and  prioritize  all  reasonable  risks.  Does  the  program 
manager  have  an  active  risk  management  program  and 
established  procedures  to  monitor  the  risks  and  update  the 
plan?  The  goal  is  to  ensure  that  the  program  manager  has, 
for  each  identified  risk,  an  integrated  mitigation  strategy. 

Dependence  on  strict  methodology  (notes,  lists,  and 
spreadsheets)  alone  for  assessing  risk  is  viewed  poorly, 
while  simple  spreadsheet  tracking  along  with  thorough  risk 
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analysis  is  viewed  more  favorably.  The  overarching  idea 
with  identifying  risk  is  that  while  a  structured  approach  is 
helpful  and  necessary,  the  very  human  input,  such  as 
thinking  between  the  lines,  uncovering  the  unexpected,  and 
an  ad  hoc  approach,  is  also  necessary  to  get  a  complete  and 
thorough  risk  assessment.  [Ref.  12] 

Besides  initial  risk  assessment  planning  and 
establishment,  another  important  factor  is  how  program 
managers  implement  it  throughout  the  program's  development. 
Is  the  Risk  Management  Plan  put  away  and  counted  merely  as  a 
data  call  satisfied,  never  to  be  used  again?  Or  is  the  Risk 
Management  Plan  implemented,  revisited,  and  updated? 

Figure  12  graphically  illustrates  the  risk  management 
hierarchy  of  the  activities  evaluated  in  the  risk  management 
component  of  the  quality  management  metric. 
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Figure  12 .  Risk  Management  Hierarchy  Factors 

Any  risk  management  plan  is  useless  unless  it  is 
updated  along  with  the  software  program' s  changing 
environment .  The  constantly  changing  environment  from 
organizational  strategy,  competitive  pressures,  changing 
political  landscape,  technical  challenges,  and  personnel 
changes,  may  dramatically  alter  a  program.  [Ref.  12] 

It  is  difficult  to  measure  individual  judgements  about 
risk  management.  What  can  be  measured  is  whether  the 
program  manager  has  performed  risk  management  elements. 


VI .  CONSTRAINT  FACTORS 


Constraints  are  factors  limiting  the  options  that  the 
program  manager  has  in  executing  the  program.  The  program 
manager's  quality  score  should  not  be  impacted  by  actions 
that  are  not  within  the  program  manager's  control.  For  a 
software  development  program,  the  two  main  sources  of 
constraints  are  those  imposed  by  the  company  or  organization 
itself  and  those  from  the  stakeholders  of  the  program  [Ref. 
12]  .  Money  and  schedule  are  typically  how  constraints  are 
imposed  [Ref.  41]  .  In  other  cases  constraints  may  be  a 
mandated  architecture,  development  model,  operating  system, 
or  suite  of  development  tools  (e.g.,  compilers,  CASE  tools, 
configuration  control,  and  management  tools) .  All  software 
development  programs  contain  constraints  that  the  program 
manager  must  content  with. 

A.  REQUIREMENTS  MANAGEMENT  CONSTRAINTS 

Constraints  in  requirements  management  include:  using 
requirements  extracted  by  other  groups,  no  control  of 
requirement  implementation,  no  prioritization  flexibility 
(all  requirements  are  priority  one) ,  and  little  to  no 
interaction  with  the  users.  One  of  the  most  frequent 
constraints  facing  program  managers  is  not  being  able  to 
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limit  requirement  changes  during  the  program  execution  [Ref. 
41]  .  Even  with  a  rigorous  change  management  structure, 
stakeholders  can  and  do  dictate  circumvention  of  the  process 
to  facilitate  their  desires  or  changing  needs  [Ref.  12]. 

B.  ESTIMATION/PLANNING  MANAGEMENT  CONSTRAINTS 

Money  and  time  are  the  main  constraint  factors  in  the 
estimation  and  planning  activities  of  a  software  development 
project  and  therefore  must  be  treated  as  resources  that  are 
to  be  managed.  Program  managers  are  often  forced  to  buy  in 
to  programs  that  are  either  inadequately  funded  and/ or  have 
unrealistic  schedules  [Ref.  12].  Frequently  money,  time,  or 
both  strictly  define  the  capabilities  of  the  software 
product  being  developed. 

Other  constraints  include  stakeholder-mandated  use  of 
specific  metrics  for  estimating.  Applying  different  metrics 
can  yield  different  estimation  results,  therefore  the 
mandated  choice  of  a  particular  metric  on  which  to  base 
estimations  can  influence  planning,  and  thus  execution  of  a 
program.  Boehm  [Ref.  1]  further  adds. 

Having  a  good  software  cost  model  available  does  not  guarantee  good 
software  cost  estimates.  As  with  other  computer-based  models,  a  software 
cost-estimation  model  is  a  “garbage  in-garbage  out”  device:  if  you  put 
poor  sizing  and  attribute-rating  data  in  on  one  side,  you  will  receive  poor 
cost  estimates  out  the  other  side. 
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The  implication  is  that  certain  types  of  software 
developments  are  better  suited  to  certain  metric  estimation 
models  than  other  programs  are.  The  program  manager  must  be 
afforded  the  opportunity  to  evaluate  alternative  techniques 
and  compare  their  relative  strengths  and  weaknesses.  [Ref. 
1] 

C.  RISK  MANAGEMENT  CONSTRAINTS 

Risk  management  constraints  primarily  involve  funding. 
Nifontoff  [Ref.  13]  states  that  risk  management  can  be  done 
cheaply  or  expensively.  The  cheap  method  is  to  rely  on  the 
existing  senior  program  managers  and  engineers  to  perform 
risk  management.  The  expensive  method  is  to  bring  in 
outside  consultants  to  perform  risk  assessment  and 
mitigation. 

Stakeholder  views  on  the  importance  of  and  willingness  to 
adopt  and  act  on  risk  management  recommendations  influence 
the  amount  of  funding  allocated  to  the  effort.  Even  if 
stakeholders  refuse  to  fund  risk  management  efforts  as  a 
separate  line  item,  Nifontoff  [Ref.  13]  says  the  program 
manager  must  perform  risk  management. 
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. .  .whether  computerized  or  with  wall  charts  or  sitting  around  a  table,  it 
still  must  be  done. 

Consequences  of  not  performing  risk  management  can  be 
devastating  to  a  software  program.  Programs  have  failed 
even  though  all  the  other  areas  were  sufficiently  addressed 
because  of  failure  to  consider  risks  [Ref  41]  . 

D.  PEOPLE  MANAGEMENT  CONSTRAINTS 

There  are  many  possible  constraint  factors  in  people 
management.  Most  of  these  involve  constraints  imposed  by 
the  company  or  organization  [Ref.  36].  The  program  manager 
may  be  unable  to  obtain  qualified  personnel  or  to  release 
team  members  who  do  not  fulfill  program  needs.  The 
limitations  on  salary  compensation,  rewards,  and  bonuses  can 
be  more  restrictive  in  Government  organizations  than 
commercial  companies  [Ref.  13] .  Executing  a  software 
development  program  within  an  activity  or  company  with  an 
organizational  structure  classified  as  a  system  one  or 
system  two  in  the  Likert  model  is  a  constraint  [Ref.  36]. 
Pickering  [Ref.  35]  believes  that  the  program  manager  must 
structure  the  software  development  team  as  a  system  three  or 
system  four  to  be  successful.  In  this  scenario,  the 
constraint  imposed  by  the  overall  organization  must  be 
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overcome.  Pickering  [Ref.  35]  adds  that  often,  whole 
organizations  change  this  way  --  from  the  bottom  up. 

The  lack  of  training  provided  by  the  organization  is 
another  constraint  in  people  management  [Ref.  36].  In  most 
organizations,  funding  for  training  is  separate  from  the 
specific  program  funding.  Therefore  the  program  manager  may 
not  have  an  ability  to  provide  needed  training  for 
individual  team  members  within  the  organization. 

E .  QUESTIONS 

Questions  exist  in  each  of  the  four  sections  that  help 
ascertain  where  program  management  may  be  constrained.  In 
the  yes-no-n/a  questionnaire,  the  not  applicable  (N/A) 
selection  is  used  for  questions  that  do  not  apply  to  the 
program  or  for  areas  in  which  the  program  manager  does  not 
have  control.  The  questions  are  designed  so  the  quality 
management  scoring  will  not  be  affected  where  constraints 
are  present . 
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VII.  METRIC  METHODOLOGY 


A.  STRATEGY 

The  approach  used  to  develop  the  metric  included 
researching  the  successful  current  and  recommended  practices 
chronicled  in  textbooks  and  periodical  publications,  and 
obtained  via  both  interviews  with  senior  program  managers 
and  conducting  focus-group  meetings.  The  metric  measures 
the  quality  of  management  on  a  specific  software  program. 
The  overall  goal  is  to  develop  an  objective,  standardized 
metric  to  which  program  management  can  be  compared  and 
ranked,  thus  providing  a  baseline  for  quantifying 
improvement .  This  metric  compares  the  same  management  on 

two  different  software  programs  or  at  two  different  time 

intervals  of  the  same  program.  Metric  development  is 

difficult  because  the  quality  of  management  can  be  very 
subjective.  Words  like  feel,  think,  believes,  etc.  which 
prompt  subjective  responses  are  avoided  as  much  as  possible. 
Subjective  answers  are  not  useful  in  obtaining  quantifiable, 
objective  results.  Answers  are  constrained  to  enable 
scoring  to  a  scale.  The  technique  used  is  a  two-part 

questionnaire  for  each  of  the  four  sections. 
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B.  QUESTIONNAIRE  FORMAT  AND  SCORING 

Questions  and  concepts  used  in  the  questionnaires  were 
gathered  and  compiled  from  periodical  articles,  textbooks, 
interviews  and  focus  groups .  The  concepts  included  are 
relevant  to  judging  the  quality  of  management.  Participants 
taking  the  survey  were  asked  to  consider  one  software 
program  at  one  particular  instance  of  time. 

Part  one  of  the  questionnaire  contains  pair  choice 
questions.  The  person  filling  out  the  questionnaire  must 
choose  one  of  the  two  statements  that  best  describe  the 
program.  The  choice  does  not  have  to  match  exactly;  it 
should  just  be  the  closest  fit.  The  model  for  this  type  of 
questionnaire  is  the  Myers-Briggs  Type  Indicator  (MBTI) 
questionnaire  [Ref.  33]  .  The  format  used  in  the  .  MBTI 
questionnaire  requires  participants  to  choose  between  two 
statements.-  Each  pair  statement  represents  two  differing 
ideas  in  an  effort  to  ascertain  a  tendency  ’  of  the 
individual .  Often  the  pair  choices  are  repeated  with 
different  wording  to  confirm  earlier  choices  and  measure  the 
strength  of  the  tendency.  The  survey  format,  with  the 
proper  mix  of  questions  and  variation  repetitions  is 
intended  to  be  used  to  reach  consensus  on  issues  and  measure 
the  strength  of  tendencies.  Each  section  has  a  maximum 


score  of  70  points.  The  risk  management, 
estimation/planning  management,  and  people  management 
sections  have  70  questions  each.  The  70  questions  in  the 
people  management  section  are  apportioned  according  to  the 
importance  rankings  of  four  of  its  lower-level  factors. 
Some  questions  apply  to  more  than  one  factor.  The 
requirements  management  section  has  50  questions  and 
includes  an  alternate  block  of  sixteen  questions  depending 
on  the  development  strategy  used. 

Part  two  of  each  questionnaire  is  the  yes-no-n/a 
questions.  Instead  of  asking  open-ended  questions  that 
participants  could  answer  in  a  variety  of  ways  in  essay 
form,  the  yes-no-n/a  format  standardizes  the  responses  for 
easier  comparison.  The  yes-no-n/a  format  is  user-friendly 
for  conducting  surveys,  requiring  minimum  writing  by  the 
participant.  Each  yes,  no,  or  n/a  choice  has  a  point  value 
based  on  the  relative  importance  of  the  question.  Each 
section  has  a  maximum  value  of  62  points.  The 
estimation/planning  management,  people  management,  and  risk 
management  sections  have  50  questions  each.  The  requirement 
section  has  47  including  an  alternative  block  of  six 
questions  depending  on  the  development  strategy  used.  The 
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complete  survey,  including  both  parts  for  all  four  sections, 
contains  457  questions. 

Administration  of  the  questionnaire  to  each  participant 
was  conducted  such  that  the  subject  was  not  given  any 
information  about  the  point  value  of  each  response;  this  was 
done  to  avoid  any  pre-bias  tendency  of  one  response  over 
another.  Manually  scoring  the  questionnaire  focuses 
attention  on  the  entire  process  and  de-emphasizes  focusing 
only  on  the  final  Quality  Management  Metric  (QMM)  score. 

The  point  totals  from  each  of  the  two  questionnaire 
parts  per  section  are  entered  on  the  QMM  Summary  Score 
Sheet.  Point  totals  for  part  one  and  part  two  are  then 
added  together  to  determine  the  total  points  for  each 
section.  The  total  points  of  each  section  are  multiplied  by 
its  relative  Importance  Coefficient  (IC)  to  yield  a  weighted 
score.  After  weighted  scores  are  determined  for  each  of  the 
four  sections,  they  are  summed  together  to  yield  the  Quality 
Management  Metric  (QMM)  score. 

The  IC  was  determined  from  the  relative  rankings  of 
importance  of  each  of  the  sections.  Experienced  software 
professionals  provided  the  data  to  determine  the  IC  through 
the  focus  groups  and  one-on-one  interviews  only  after 
thorough  explanation  and  understanding  of  each  category.  A 
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total  value  of  forty  points  was  allowed  for  allocation  over 
the  four  sections. 

Figure  13  is  the  summary  of  the  raw  data  used  to 
determine  each  section  IC.  The  QMM  equation  is  as  follows: 
QMM  =0.92  RqM  +  0.67  EPM  +0.55  RkM  +  1.86  PM 
The  QMM  is  the  sum  of  four  components: 

RqM  is  the  requirements  management  metric 
EPM  is  the  estimation/planning  metric 
RkM  is  the  risk  management  metric 
PM  is  the  people  management  metric 

Because  of  the  overwhelming  importance  placed  in  people 
management,  PM  is  further  broken  into  four  components  that 
were  individually  ranked.  The  PM  is  the  sum  of  its  four 
components , 

The  four  components  are  L,  the  leadership  measure,  C, 
the  communication  measure,  HR,  the  human  resource  measure, 
and  TC,  the  technical  competency  measure.  Data  for 
determining  the  IC  in  each  of  the  four  components  of  people 
management  was  gathered  with  the  same  methods  used  to 
determine  the  IC  for  the  four  management  sections.  However, 
the  total  points  spread  across  the  people  management 
components  could  not  exceed  the  total  points  allocated  for 
people  management. 
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The  equation  for  People  Management  is  PM  =  0.65  L  + 
0.55  C  +  0.41  HR  +  0.25  TC. 


RATED 
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Figure  13 .  Importance  Coefficient  Development 

The  maximum  QMM  score  possible  for  the  entire  survey  is 
528.00  points  and  the  minimum  possible  score  is  -130.86  as 
part  two  questionnaires  contain  negative  point  response 
values . 
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VIII.  INFORMAL  VERIFICATION  AND  VALIDATION 


A.  MOTIVATION 

The  structure  and  methodology  for  evaluating  the 
quality  of  software  management  laid  out  in  the  previous 
chapters  is  informally  verified  and  validated  in  this 
section.  The  informal  verification  and  validation  is 
necessary,  to  ensure  that  the  metric  measures  the  quality  of 
software  program  management  and  that  it  does  so  as 
accurately  as  possible. 

B .  STRATEGY 

The  approach  to  verification  and  validation  is 
informal .  Three  software  programs  were  evaluated  for  a  QMM 
score.  The  program  manager  and  one  program  development  team 
member  evaluated  program  A.  Program  B  was  evaluated  by  the 
program  manager  and  two  program  development  team  members, 
and  program  C  was  evaluated  by  the  program  manager  and  one 
program  team  member. 

In  order  to  provide  a  frame  of  reference  in  which  to 
correlate  initial  survey  results,  two  other  measures  were 
developed  and  used.  These  two  measures  are  the  QMM 
percentage  score  and  the  overall  program  success  score. 
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The  QMM  percentage  score  is  a  derived  measure  of  the 
QMM  score.  To  obtain  a  QMM  percentage  score,  the  following 
steps  are  required.  First,  the  survey  minimum  possible 
score  is  normalized  to  zero.  Since  the  survey  minimum  QMM 
score  possible  is  -130.86,  130.86  is  added  to  the  survey 

minimum  possible  score  in  order  to  have  it  equal  zero. 
Correspondingly,  130.86  must  be  added  to  both  the  survey 
maximum  QMM  score  possible  and  to  the  actual  QMM  score 
obtained  in  the  survey.  Since  the  QMM  survey  maximum 
possible  score  is  528.00,  the  resulting  normalized  survey 
maximum  possible  score  is  658.86. 

To  obtain  a  QMM  percentage  score,  the  normalized  QMM 
score  obtained  from  the  survey  is  divided  by  the  normalized 
survey  maximum  possible  QMM  score  and  then  multiplied  by 
100.  Thus,  the  equations  are  as  follows: 

QMM^jjj  +  130.86  =  0.00  =  QMMj^m  j,ophali2ed 

QMMu^^  +  130.86  =  658.86  =  QMMj^^^  normalized 

QMMgcoRE  +  130.86  =  QMM3j,Qjjg  j,qj(j^j2ed 

( QMMgcQRg 

NORMALIZED  normalized)  *100-QMMpERCENTAGE  SCORE 

The  overall  success  score  is  a  subjective  number 
assigned  by  the  survey  participant  rating  the  overall 
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success  of  the  program.  The  success  of  a  program  is 
measured  in  terms  of  how  well  the  final  product  performs  and 
meets  the  expectation  and  satisfaction  of  users  and 
stakeholders . 

The  survey  participant's  QMM  score  is  compared  to  his 
or  her  individual  overall  success  score  and  to  the  mean 
overall  success  score  of  the  program.  The  mean  overall 
success  score  of  a  program  is  derived  from  each  survey 
participant's  individual  overall  success  score  and  at  least 
two  other  individuals  (mostly  users,  or  those  somehow 
associated  with  the  program  or  delivered  product)  able  to 
judge  the  overall  success  of  the  program. 

The  overall  success  score  is  measured  on  a  scale  of 
zero  to  ten.  Zero  is  defined  as  abject  program  failure  with 
no  worthwhile  product.  Ten  is  defined  as  absolutely  perfect 
software  product  associated  with  flawless  program  execution. 
The  cause  for  success  or  failure  of  the  program  is  not 
important .  It  may  or  may  not  be  associated  with  any  actions 
involving  program  management. 

The  QMM  percentage  score  is  compared  to  the  individual 
and  mean  overall  success  score  of  the  program. 

The  goal  was  to  determine  any  correlation  between  the 
participants'  QMM  score,  their  individual  success  ranking  of 
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the  overall  program,  and  the  mean  success  ranking  of  the 
overall  program.  For  example,  an  overall  success  score 
seven  corresponding  to  a  QMM  percentage  score  of  70  percent 
plus  or  minus  5  percent  indicates  strong  correlation.  An 
overall  success  score  of  seven  corresponding  to  a  QMM 
percentage  score  greater  than  plus  or  minus  five  percentage 
points  of  70  percent,  but  less  than  plus  or  minus  15 
percentage  points  of  70  percent  is  considered  fair 
correlation.  If  a  program  has  an  overall  success  score  of  8 
corresponding  to  a  QMM  percentage  score  of  40  percent,  this 
would  be  considered  weak  correlation.  In  this  last  case, 
the  QMM  metric  is  still  valid  as  programs  with  high  quality 
of  software  program  management  could  conceivably  fail  for  a 
variety  of  reasons,  including  poor  technology  or  funding 
shortfalls.  The  reverse  condition  may  also  be  true  for 
explaining  successful  programs  with  low  quality  of  software 
management.  However,  it  is  typically  expected  that 
successful  software  programs  follow  superior  software 
program  management . 
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C .  RESULTS 

After  the  survey  was  scored,  a  QMM  was  determined  for 
the  program.  The  QMM  score  is  measured  as  a  percentage  of 
the  maximum  QMM  score  possible.  That  percentage  was 
compared  to  the  subjective  assigned  score  of  the  relative 
success  of  the  project  to  obtain  a  comparison  basis.  Table 
1  summarizes  the  resultant  scores  of  the  three  programs. 
The  subscript  PM  indicates  the  program  manager's  survey 
results  and  the  subscript  numeral  indicates  a  participant's 
survey  results  other  than  the  program  manager.  The  mean 
success  score  of  a  program  includes  the  individual  success 
ranking  scores  by  the  individuals  participating  in  the 
survey  plus  others  associated  with  the  program  in  some  way 
where  they  can  judge  the  success  of  the  program. 


Program 

Program  B 

Program  C 

Participant 

A, 

BpM 

CpM 

Cx 

QMM  Score 

338 

106 

47 

198 

189 

QMM  percent 

71.2% 

68.8 

35.9% 

27.0% 

49.9% 

48.6% 

Success  score 

7 

7 

9 

4 

3 

4 

4 

Mean  success 
score  (0-10) 

7 

4 

4 

Table  1 .  Results  Summary  Comparison 

The  survey  results  for  program  A  and  program  C  exhibit 
correlation  between  the  QMM  percentage  ranking  and  the 
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overall  success  ranking  of  the  program,  both  with  individual 
success  ranking  scores  and  the  mean  ranking  score.  The  QMM 
summary  sheets  for  each  survey  completed  are  enclosed  as 
Appendix  A.  An  examination  of  the  summary  sheets  for 
program  A  reveals  a  weak  risk  management  section.  This 
conclusion  appears  correct,  as  risk  management  for  this 
program  was  not  emphasized.  However,  program  A  was  highly 
structured  and  planned,  involved  key  users  well  enough  to 
warrant  successful  requirement  extraction  and  enjoyed  good 
technical  success  with  their  deliverables.  The  higher 
scores  in  the  other  three  sections  reflect  this  success. 

Program  C  was  a  smaller  program  that  was  relatively 
unstructured,  with  essentially  no  risk  management,  little 
planning  and  poor  requirement  extraction.  However,  the 
program  has  delivered  a  usable  product,  primarily  on  the 
heels  of  strong  practices  in  the  people -management  portion 
and  a  technology  that  was  relatively  straightforward  and 
understood. 

Program  B  exhibits  a  significant  divergence  from  the 
scores  of  the  program  manager  and  the  other  team  members . 
In  this  case  the  program  manager's  scores  on  both  the  QMM 
and  individual  success  ranking  were  significantly  greater 
than  the  mean  success  ranking  and  the  QMM  scores  and 
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individual  rankings  of  the  other  two  participants  from 
program  B.  This  program  appears  to  have  a  dichotomy  in 
perception.  Further  interviews  with  others  in  the  program 
and  users  of  the  product  indicate  that  there  are  some 
significant  issues  needing  resolution. 

Having  a  good  method  or  model  does  not  guarantee  good 
results  [Ref.  1] .  Inaccurate  or  incomplete  information  will 
significantly  affect  the  validity  of  '  survey  scores. 
Additionally,  the  self -enhancement  bias  is  a  peirverse  social 
psychological  phenomenon.  Researchers  have  found  that  one 
of  the  most  widely  documented  effects  in  social  psychology 
is  the  preference  of  most  people  to  see  themselves  in  a 
self -enhancing  fashion  [Ref.  25]  .  On  the  job,  approximately 
90  percent  of  managers  and  workers  rate  their  performance 
superior  to  that  of  their  peers  [Ref.  42]  .  Surprisingly,  it 
is  not  only  the  answers  to  the  more  subjective  survey 
questions  that  vary  among  participants  in  the  same  program, 
it  is  also  some  of  the  answers  to  the  purely  objective 
questions  on  the  survey.  These  results  not  only  make  the 
case  for  the  requirement  of  a  survey  administrator;  it  also 
points  to  a  need  for  conducting  interviews  in  addition  to 
administering  the  survey  to  better  judge  the  results. 
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Survey  results  that  vary  significantly  between  program 
management  and  team  members  can  be  very  useful  in  uncovering 
significant  differences  in  perceptions  about  what  is  thought 
to  be  occurring  and  required  in  a  program  and  what  is 
actually  occurring  and  required  in  the  program.  Bringing 
the  participants  together  after  the  survey  has  been 
completed  and  scored  to  discuss  the  significant  differences 
in  their  answers  could  be  the  single  biggest  benefit  of  the 
QMM  process . 

The  participants  provided  additional  feedback  and 
recommendations  for  improvements  to  the  concepts  surveyed  in 
each  of  the  sections  and  also  for  improvements  in  individual 
questions  asked.  Copies  of  the  QMM  summary  sheets  for  all 
seven  of  the  survey  participants  are  included  in  Appendix  A. 
Copies  of  the  completed  survey  from  each  of  the  three 
program  managers  are  included  in  Appendix  B .  The  resultant 
survey  questionnaire  template  with  ranking  of  each  response 
is  included  as  the  Appendix  C. 
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IX.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

This  thesis  provides  an  initial  structure  and  basis  for 
evaluating  software  management  performance  on  specific 
software  programs.  The  goal  of  creating  an  objective, 
repeatable  metric  for  determining  the  quality  of  software 
management  was  obtained.  The  quality  of  management  on 
software  programs  varies  considerably  and  is  a  significant 
element  in  the  overall  success  of  a  program  [Ref.  1]  .  The 
policies  and  decisions  that  the  program  managers  make 
influence  the  success  of  a  software  program. 

1.  Top-Level  Evaluation  Sections 

Individual  software  program  managers  vary  in  their 
style  of  running  a  program.  Using  the  MBTI  as  a  model,  the 
thesis  identified  important  characteristics  of  successful 
managers  and  rated  them  accordingly  via  the  two-part 
questionnaire.  The  four  top-level  evaluation  sections, 
requirements  management,  estimation/planning  management, 
people  management,  and  risk  management  encompass  all 
activities  and  techniques  used  to  execute  a  software 
program.  Overwhelmingly,  the  people -management  section  was 
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rated  highest  in  importance  in  judging  the  software  program 
manager's  performance.  Four  lower- level  factors, 
leadership,  communication,  human  resources,  and  technical 
competency  of  the  program  manager  were  subsequently 
individually  categorized  and  rated  within  the  people 
management  section  alone.  Focus  groups  and  survey 
participant's  results  concurred  that  the  people  management 
factor  dominates  a  software  program  manager's  performance. 

2 .  Survey 

The  survey  format,  length,  and  individual  questions 
achieved  the  goal  of  covering  the  important  processes  and 
concepts  involved  in  the  quality  of  the  software  manager  in 
an  acceptable  amount  of  time  dedicated  by  the  participants. 
The  average  survey  completion  time  was  approximately  45 
minutes.  The  longest  timed  participant  took  almost  60 
minutes  and  the  shortest  '  timed  participant  took 
approximately  35  minutes. 

3 .  Metric  Scoring 

The  comparison  of  the  QMM  percentage  score  to  each 
individual  overall  program  success  score  yielded  strong 
correlation  in  each  instance.  The  comparison  of  the  QMM 
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percentage  score  to  the  mean  overall  program  success  score 
yielded  strong  correlation  in  all  but  one  instance. 

Six  of  the  seven  survey  participants  recorded  QMM 
percentage  scores  within  13  points  of  the  mean  success  score 
percentage  for  their  respective  programs.  This  indicates 
strong  correlation  of  the  metric  with  program  performance. 

The  one  exception  was  the  highest  QMM  score  recorded  at 
386.15  and  with  a  corresponding  QMM  percentage  score  of 
78.5%  on  a  program  with  a  mean  success  rating  of  4  exhibited 
a  significant  variance.  However,  that  participant  gave  that 
same  program  an  individual  program  success  score  of  9,  which 
was  also  a  significant  variance  from  the  mean  success  score 
of  4.  This  divergence  indicates  a  significant  difference  in 
perception  of  the  program  and  program  management .  Since 
this  survey  result  was  from  a  program  manager,  at  least  two 
plausible  explanations  may  exist.  Either  the  program 
manager  is  answering  the  survey  as  how  he  thinks  the  program 
should  be  run  as  opposed  to  how  it  is  actually  is  run  or  the 
processes  and  structure  the  program  manager  has  established 
for  the  program  are  not  understood  well  enough  by  other 
development  team  members . 
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B .  RECOMMENDATIONS 

Using  the  survey  questionnaires  as  a  guide,  program 
management  performance  can  be  improved  by  evaluating 
questions  where  choices  selected  were  not  scored  as  the 
preferred  alternative.  Participants  taking  the  survey  for 
the  same  program  over  the  same  timeframe  can  uncover 
significant  dichotomies  when  discussing  questions  where  the 
responses  were  different. 

1.  Top-Level  Evaluation  Sections 

Software  engineering  is  not  a  static  discipline.  New 
techniques  and  improved  strategies  are  constantly  being 
developed.  Further  re-evaluation  of  the  lower  factors  in 
each  of  the  top-level  factor  sections  can  serve  to  refine 
the  basis  for  evaluating  the  quality  of  software  program 
management . 

While  the  QMM  score  can  give  the  program  manager  a  view 
of  past  and  present  performance,  reviewing  the  questions  in 
factor  sections  where  scores  are  weaker  can  provide  insight 
and  guidance  to  the  software  program  manager  seeking 
improvement.  The  survey  is  intended  to  be  administered  by 


individuals 

who 

understand 

the 

elements,  motivation. 

and 

scoring  of 

the 

questions 

and 

responses  in  each  of 

the 
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sections.  These  administering  individuals  can  then  provide 
one-on-one  guidance  and  further  explanation  to  the  software 
program  manager  throughout  the  process.  Additionally, 
survey  administrators  should  interview  the  survey 
participants  to  uncover  any  pre-bias  or  misperceptions  that 
may  exist  and  influence  the  survey  results. 

2 .  Survey 

As  new  techniques  and  improved  strategies  are 
developed,  the  questionnaires  must  be  continually  refined  to 
assure  that  higher  scores  relate  to  higher  software 
management  performance .  Immediate  future  work  should  focus 
on  refining  the  questions  in  each  of  the  questionnaires  to 
better  reflect  desired  outcomes  of  software  programs.  This 
can  be  accomplished  in  three  ways. 

The  first  way  is  to  improve  the  wording  of  existing 
questions  to  improve  the  clarification  of  concepts  and  to 
eliminate  wording  that  could  imply  a  preferred  response.  If 
the  appearance  of  response  choices  is  neutral  there  is  less 
likely  a  temptation  of  the  survey  participant  to, 
consciously  or  subconsciously,  choose  the  implied  correct 
response  rather  than  the  appropriate  response  reflecting 
current  conditions  in  the  program. 


79 


Secondarily,  survey  improvement  may  be  attained  from 
formulation  of  replacement  questions.  The  attempt  would  be 
to  adjust  focus  on  the  more  important  concepts  that 
determine  software  management  quality. 

Finally,  refining  the  point  values  of  individual 
responses  can  improve  the  survey.  Responses  for  the  more 
important  concepts  must  be  reflected  with  higher  point 
values  than  responses  given  for  questions  more  marginal  in 
determining  software  management  quality. 

Based  on  feedback  from  survey  participants,  the  current 
length  of  the  survey  is  appropriate  for  coverage  of  the 
material  important  in  evaluating  software  management. 
However,  any  increase  in  length  of  the  survey  was  viewed  as 
a  negative  and  would  discourage  participation.  Therefore, 
the  emphasis  for  improvement  in  the  questionnaires  must  be 
from  refinement  and  replacement  of  current  questions. 

3 .  Metric  Scoring 

This  thesis  provides  an  informal  verification  and 
validation,  evaluating  only  three  programs  for  a  QMM  score. 
All  three  programs  were  Department  of  Defense  systems.  A 
greater  number  of  software  program  managers  and  key  team 
members,  in  addition  to  a  greater  variety  of  software 
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programs,  need  to  be  evaluated  to  establish  a  statistically 
significant  correlation  of  the  QMM  score  to  overall  software 
program  success.  The  process  is  iterative  and  may 
necessitate  adjustment  in  scoring  the  metric  to  better 
correlate  with  software  program  performance.  Particular 
attention  should  be  noted  for  programs  of  various  sizes  and 
types.  Metric  scoring  formulation  may  require  different 
coefficients  based  on  whether  the  software  development  is 
commercial  or  government .  Metric  scoring  may  also  require 
different  coefficients  based  on  the  size  or  complexity  of 
software  developments.  These  conclusions  can  only  be 
attained  after  significant  numbers  of  surveys  are  conducted 
and  their  results  evaluated  for  statistically  significant 
trends . 

As  additional  surveys  are  conducted,  the  collective 
understanding  of  what  constitutes  the  quality  of  software 
program  management  will  continue  to  grow.  Applying 
measurement  to  the  quality  of  software  management  will  lead 
to  improvements  of  program  managers  and  the  likelihood  of 
the  success  and  quality  of  their  software  programs. 
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Program:  C  QMM  Summary  Score  Sheet  Date:  Nov  99 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements  Management 

a 

48 

e 

34 

82 

X 

0.92 

75.73 

Est./Planning  Management 

b 

50 

f 

38 

88 

X 

0.67 

= 

59.01 

People  Management 

c 

48 

51 

99 

1.86 

184.61 

Risk  Management 

33 

1 

34 

0.55 

= 

18.60 

QMM  SCORE  337.95 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  71.15% 

Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  A-pm 
Success  Score;  7 
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Program:  C 


QMM  Summary  Score  Sheet 


Date:  Nov  99 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements  Management 

a 

44 

e 

35 

79 

0.92 

s: 

72.96 

Est./Planning  Management 

43 

26 

69 

B 

0.67 

s 

46.27 

People  Management 

c 

54 

45 

99 

I 

1.86 

= 

184.61 

Risk  Management 

£ 

33 

1 

34 

1 

0.55 

18.60 

QMM  SCORE  322.44 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  68.80% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  1 0  being  perfect  program  total  success) 

Survey  Participant:  A-1 
Success  Score:  7 
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Program:  C 


QMM  Summary  Score  Sheet 


Date:  Nov  99 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements  Management 

a 

42 

e 

39 

81 

D 

0.92 

74.81 

Est./Planning  Management 

b 

57 

36 

93 

B 

0.67 

a 

62.36 

People  Management 

c 

58 

50 

108 

1.86 

B 

201.39 

Risk  Management 

£ 

44 

h 

43 

87 

1 

0.55 

R 

47.59 

QMM  SCORE  386.15 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  78.47% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  B-pm 
Success  Score:  7 
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Program:  C 


QMM  Summary  Score  Sheet 


Date:  Nov  99 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

importance 

■ 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

1 

Score 

Requirements  Management 

a 

29 

e 

13 

42 

D 

0.92 

1 

38.79 

Est./Planning  Management 

b 

19 

£ 

-13 

6 

0.67 

1 

4.02 

People  Management 

c 

21 

5 

26 

1.86 

= 

48.48 

Risk  Management 

d 

17 

jT^ 

9 

26 

1 

0.55 

= 

14.22 

QMM  SCORE  105.52 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  35.88% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  1 0  being  perfect  program  total  success) 

Survey  Participant:  B-1 
Success  Score:  7 
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Program:  C 


QMM  Summary  Score  Sheet 


Date:  Nov  99 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

1 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements  Management 

a 

16 

e 

6 

22 

B 

0.92 

B 

20.32 

Est/Planning  Management 

b 

21 

f 

-16 

5 

0.67 

1 

3.35 

People  Management 

c 

25 

-10 

15 

1 

1.86 

27.97 

Risk  Management 

£ 

6 

h 

-15 

-9 

0.55 

= 

-4.92 

QMM  SCORE  46.72 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  26.95% 

Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  B-2 
Success  Score:  7 


88 


Program:  C 


QMM  Summary  Score  Sheet 


Date:  Nov  99 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

■ 

k  Vi  H  f » 1 

Category 

Score 

Score 

Score 

Coefficient 

Requirements  Management 

a 

23 

e 

1 

24 

B 

0.92 

1 

22.16 

Est./Pianning  Management 

b 

11 

-20 

-9 

I 

0.67 

1 

-6.04 

People  Management 

c 

52 

48 

100 

X 

1.86 

186.47 

Risk  Management 

£ 

12 

-21 

-9 

0.55 

= 

-4.92 

QMM  SCORE  197.68 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  49.86% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  1 0 
(0  being  total  failure,  1 0  being  perfect  program  total  success) 

Survey  Participant:  C-pm 
Success  Score:  7 
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Program:  C 


QMM  Summary  Score  Sheet 


Date:  Nov  99 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements  Management 

a 

29 

e 

7 

36 

B 

0.92 

z; 

33.25 

Est./Planning  Management 

18 

5 

23 

1 

0.67 

15.42 

People  Management 

c 

37 

43 

80 

B 

1.86 

= 

149.18 

Risk  Management 

£ 

7 

h 

-23 

-16 

1 

0.55 

s: 

-8.75 

QMM  SCORE  189.09 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  48.56% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  1 0 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  C-1 
Success  Score:  7 
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Requirements  Management  (page  1  of  2)  score  36 
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Pair  choice  section  TWO;  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


o^ 

a\ 

> 

o 

Z 

b' 

d 

o 


o 

a> 

GO 

C3 

Ph 


(U 

e 

CO 

12; 


bO 

O 


X  XXX 


s  i « 

>2  2  « 

®  § 

^  ’S  S 

^  i-s 

1  •§  •“ 

S  -2  &> 

2  5;= 

fO  ^  o 

-S  2  5^ 
2^0 
«  s 

M  t  •§ 

.1  i  r 

-P  M  T? 
CO  -O  g 

:3  S2 


o  CO 

s  ^ 

_n 

5  S  “ 

X  ^  *0 
^  ^  c 

O  M 


Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
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Program  Name:  A  YES-NO-N/A  Questionnaire 

No.  Requirements  Management  Questionnaire _ 


1  PM  chose  to  have  a  formal  requirements  list _ _ 


2  Requirements  recorded  in  some  way _ _ 


3 1  Written  requirements  were  part  of  some  formal  document _ 


Written  requirements  were  informal _ 


5  At  least  some  requirements  were  oral  only _ _ 


6  AH  stakeholders  were  identified  _ _ 


Ail  stakeholders  participated  in  the  requirements  extraction _ 


8 1  Some  stakeholders  participated  in  the  requirements  extraction _ 


9 1  Management  extracted  requirements,  no  stakeholder  involvement 


10| Management  passed  requirements  to  development  team _ 


1 1  Stakeholders  not  involved  in  Management  extraction,  but  approves 


12  Management  gets  inputs  from  stakeholders,  then  develops  requirments 


13  Developers  work  informally  with  users  to  arrive  at  requirements _ 


14 [Same  as  13,  but  management  oversees  and  formalizes 


Date:  November  1999 

Yes  No  N/A 


If  a  waterfall  or  sequential  development  strategy:  _ 


All  requirements  complete  before  design _ 


Some  requirements  left  incomplete  prior  to  design _ 


Requirements  informal  prior  to  design  effort _ 


Requirements  serve  as  input _ _ 


Length  of  time  for  requirements  work  greater  than  development  work 


Requirements  developed  in  parallel  to  design _ 


OR  If  a  prototype,  throwaway,  or  other  development  strategy: _ 


Learn  about  requirements  through  development  efforts _ 


No  coding  until  all  requirements  are  defined _ 


Requirements  formal  prior  to  design  effort _ 


18iRequirements  serve  as  output _ 


Requirements  definition  work  in  parallel  to  development  efforts 


Requirements  developed  in  parallel  to  design 


21  lAre  requirements  frozen  at  some  phase _ 


22  Change  management  exists _ 


23  Change  management  is  formal _ 


24  Project  strategy  is  consistent  throughout  development _ 


25  Requirements  are  updated _ 


26 1  Configuration  Management  (CM)  exists _ 


27  CM  is  formal  _ 


Requirements  are  testable _ 


Requirements  testing  considered/implemented  during  extraction 


Requirements  testing  plan  exists _ 


31 1  Requirements  testing  is  forma! _ 


32  AH  requirements  have  priorities _ 


33  All  requirements  must  be  implemented _ 


34 1  Requirements  are  tested _  _ 


35 1  All  requirements  are  equally  important 


36  At  least  some  requirements  have  priorities _ 


37  All  requirements  are  traceable _ 


38  Traceability  not  important _ 


39  Each  requirement  has  an  author _ 


Who  authored  requirement  is  not  important _ 


41 1  Initial  set  of  requirements  to  be  implemented,  no  requirements  creep _ 


42  Structured  and  tracked  changes  to  requirements  only _ 


43  Change  is  inevitable,  changes  allowed  at  all  times _ 


44  Change  is  inevitable,  but  changes  limited _ 


45  Requirements  control  funding _ 


46  Requirements  history  kept _ 


47 1  Baseline  established  for  requirements  at  some  point  prior  to  develop _ 
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Program  Name:  A  YES-NO-N/A  Questionnaire  Date:  November  1999 

No.  Estimation/Pianning  Questionnaire  Yes  No  N/A 


1 

A  volume  product  metric  used  (LOG,  #  of  files,  #  of  screens,  pages  of  doc) 

X 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

X 

3 

Product  measures  made  by  phase  (amt  at  implementation,  log  changed  at  unit  test) 

X 

4 

other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

X 

5 

Product  metrics  tracked  and  updated  throughout  program  execution 

X 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

X 

7 

Time  measure  process  metric  used  (cycle  time) 

X 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

X 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

X 

10 

Program  cost  estimations  tracked  and  updated  to  reflect  progress/changes 

X 

11 

Factor  analysis  performed  on  program 

X 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

X 

13 

Work  breakdown  structure  developed 

X 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

X 

15 

Schedules  developed  based  on  realistic  expectations 

X 

16 

Schedules  tracked  and  updated  based  on  new  information 

X 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

X 

18 

Quality  assurance  plan  or  similar  to  aid  in  detecting  defects  early  in  program 

X 

19 

COCOMO  estimates  performed 

X 

20 

CSCI  clearly  defined  and  tasked 

X 

21 

Estimates  completed  ad  hoc 

X 

22 

Gantt  charts  used  and  updated 

X 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

X 

24 

Earned  value  established 

X 

25 

Earned  value  tracked  throughout  program 

X 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

X 

27 

Critical  path  for  program  tasks  developed  and  tracked 

X 

28 

Meaure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

X 

29 

Estimates  are  updated  routinely 

X 

30 

Schedules  are  updated  routinely 

X 

31 

Estimations  are  made  by  program  management  (top-down) 

X 

32 

Estimations  are  made  by  program  team  members  (bottom-up) 

X 

33 

Automated  program  tracking  used 

X 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

X 

35 

WBS  developed  only  as  data  call 

X 

36 

Earned  value  used  to  track  program  progress 

X 

37 

PM  insists  on  prioritizing  work  reduction  as  scheduie/funding  compromised  by  stakeholders 

X 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

X 

39 

All  program  team  members  involved  in  planning  process 

X 

40 

Hardware  also  considered  in  estimation  process 

X 

41 

Program  history  compiled 

X 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

X 

43 

Management  duties  apart  of  each  team  member's  responsibilities 

X 

44 

PM  dictates  schedules  to  program  team 

X 

45 

Code  reviews  planned  in  schedule 

X 

46 

Defined  tangible  milestones  established  for  program  tasks 

X 

47 

Test  planning  done  at  the  start  of  the  program 

X 

48 

Estimations  are  completed  by  those  performing  the  tasks 

X 

49 

Sensitivity  analysis  performed  for  program  choices 

X 

50 

Software  deployment  planning  completed 

X 

■11 
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Program  Name:  A  YES-NO-N/A  Questionnaire 

No.  People  Management  Questionnaire 


Date:  November  1999 

Yes  No  N/A 


1 

PM  is  accessable  in  person  by  each  team  member 

X 

2 

PM  is  accessable  via  email  (memo,  letter)  by  each  team  member 

X 

3 

PM  is  accessable  via  phone  by  each  team  member 

X 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

X 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

X 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

X 

7 

PM  soiicits  opinions  from  team  members  before  making  decisions 

X 

8 

PM  lets  teams  make  decisions  affecting  their  work 

X 

9 

PM  frequentiy  makes  decisions  without  any  consultation  with  members 

X 

10 

PM  understands  the  technology/language  of  the  program 

X 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

X 

12 

PM  prioritizes  problems  or  conflicts  within  the  program 

X 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

X 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

X 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

X 

16 

PM  specifically  assigns  work  to  each  program  member 

X 

17 

PM  sets  communication  protocol 

X 

18 

PM  allows  unrestricted  communications 

X 

19 

PM  encourages  or  requires  training  for  each  individual 

X 

20 

PM  takes  control  in  difficult/  problem  areas 

X 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

X 

22 

PM  maintains  regular  communications  with  all  stakeholders 

X 

23 

PM  maintains  regular  communications  with  users 

X 

24 

PM  encourages  program  team  communication  with  users 

X 

25 

PM  encourages  program  team  communication  with  stakeholders 

X 

26 

PM  facilitates  horizontal  communication  within  program 

X 

27 

PM  facilitates  communication  during  integration 

X 

1 

28 

PM  holds  meetings  without  clear  objectives 

X 

29 

PM  must  approve  all  decisions  within  the  program 

X 

30 

PM  must  approve  all  interactions  with  stakeholders 

X 

31 

PM  must  approve  all  interactions  with  users 

X 

32 

PM  makes  all  presentations  to  stakeholders/users 

X 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

X 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

X 

35 

PM  is  readily  willing  to  listen  to  program  problems  and  complaints 

X 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

X 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

X 

~ 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

39 

PM  recruits  program  team  members  from  outside  organization 

X 

40 

PM  participates  in  technical  reviews 

X 

41 

Program  personnel  have  clearly  defined  specific  tasks 

X 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

X 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

X 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

X 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

X 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

X 

47 

PM  clearly  separates  technical  from  managerial  roles  for  individuals 

X 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

X 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

X 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

X 

■1 
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Program  Name:  A 


YES'NO-N/A  Questionnaire 


Date:  November  1999 


No.  Risk  Management  Questionnaire _ Yes  No  N/A 


1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

X 

2 

RM  is  formal  and  documented 

X 

3 

A  specific  RM  plan  exists 

X 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

X 

5 

RM  is  done  prior  to  the  program  execution 

X 

6 

RM  is  done  by  an  outside  entity  to  the  development 

X 

7 

RM  is  done  internally  only 

X 

8 

RM  is  both  internally  performed  and  externally  assessed 

X 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

X 

10 

Risk  Assessment  is  only  a  management  function 

X 

11 

RM  is  informal  or  non  existent 

X 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

X 

13 

Risks  are  only  generalized 

X 

14 

Each  risk  is  delineated 

X 

15 

Each  risk  has  a  consequence 

X 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

X 

17 

Each  risk  has  a  mitigation  strategy 

X 

18 

Risk  Management  is  automated 

X 

19 

Risks  are  tracked 

X 

21 

Regret  analysis  performed 

X 

22 

RM  drives  decisions  in  the  program 

X 

23 

Risks  have  probabilities 

X 

24 

Risk  Management  is  ad  hoc 

X 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

X 

26 

Risks  are  weighed  relative  to  other  program  risks 

X 

27 

Risk  Assessment  is  a  program  team  activity 

X 

28 

Risk  Assessment  done  prior  to  program  start 

X 

29 

Risk  Assessment  includes  personnel  risk 

X 

30 

RM  uses  tools,  but  depends  on  human  decisions 

X 

31 

Risk  Assessment  includes  cost  risks 

X 

32 

Risk  Assessment  includes  schedule  risks 

X 

33 

Risk  Assessment  includes  technology  risks 

X 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

X 

35 

Risk  Assessment  includes  requirements  risks 

X 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

X 

37 

Risk  Assessment  includes  documentation  risks 

X 

38 

Risk  Assessment  includes  integration  risks 

X 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

X 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

X 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

X 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

X 

43 

High  risk  have  measured  tracking  (high  profile  status) 

X 

44 

Organizational  history  used  to  search  for  risks 

X 

45 

Other  organizational  checklists  used  for  risk  assessment 

X 

46 

Internal  organizational  checklists  used  for  risk  assessment 

X 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

X 

48 

Risk  Assessment  includes  internal  organization  risks 

X 

49 

Risk  Assessment  includes  stakeholder  risks 

X 

50 

No  risk  management  needed;  program  is  straightforwarded  &  understood 

Eai 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
Leadership: _ 
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Program  Name:  B  YES-NO-N/A  Questionnaire 

No.  Requirements  Management  Questionnaire  _ 

1 1  PM  chose  to  have  a  formal  requirements  list 

_ 2  Requirements  recorded  in  some  way _ 

_ 3  Written  requirements  were  part  of  some  formal  document _ 

4  Written  requirements  were  informal _ _ 

_ 5  At  least  some  requirements  were  oral  only _ _ 

_ 6  All  stakeholders  were  identified _ _ _ 

_ 7  All  stakeholders  participated  in  the  requirements  extraction _ 

8  Some  stakeholders  participated  in  the  requirements  extraction _ 

9  Management  extracted  requirements,  no  stakeholder  involvement 

1 0  Management  passed  requirements  to  development  team _ 

1 1  Stakeholders  not  involved  in  Management  extraction,  but  approves 

12  Management  gets  inputs  from  stakeholders,  then  develops  requirments 

13  Developers  work  informally  with  users  to  arrive  at  requirements 

14  Same  as  13,  but  management  oversees  and  formalize 


Date:  Nov  99 


Yes  No  N/A 

nn  ^  I 


If  a  waterfall  or  sequential  development  strategy: _ 


15  All  requirements  complete  before  design _ 


lelSome  requirements  left  incomplete  prior  to  design _ _ 


Requirements  informal  prior  to  design  effort _ _ 


Requirements  serve  as  input _ 


Length  of  time  for  requirements  work  greater  than  development  work 


Requirements  developed  in  parallel  to  design _ _ 


OR  If  a  prototype,  throwaway,  or  other  development  strategy: _ 

1  Learn  about  requirements  through  development  efforts 


No  coding  until  all  requirements  are  defined _ _ 


Requirements  formal  prior  to  design  effort _ _ 


Requirements  serve  as  output _ 


Requirements  definition  work  in  parallel  to  development  efforts 


Requirements  developed  in  parallel  to  design _ 


21 1  Are  requirements  frozen  at  some  phase _ 


22  Change  management  exists _ 


23 1  Change  management  is  formal _ 


24  Project  strategy  is  consistent  throughout  development 


25  Requirements  are  updated _ _ _ 


26  Configuration  Management  (CM)  exists _ 


27  CM  is  formal  _ 


Requirements  are  testable _ 


Requirements  testing  considered/implemented  during  extraction 


Requirements  testing  plan  exists _ _ _ 


31  [Requirements  testing  is  forma! _ _ _ 


32  All  requirements  have  priorities _ _ 


33  All  requirements  must  be  implemented _ _ 


34  Requirements  are  tested _ _ _ 


35  All  requirements  are  equally  important _ _ 


36  At  least  some  requirements  have  priorities _ _ _ 


37 1  All  requirements  are  traceable _ _ _ _ 


Traceability  not  important _ _ _ 


39 1  Each  requirement  has  an  author _ _ _ _ 


Who  authored  requirement  is  not  important _ 


41 1  Initial  set  of  requirements  to  be  implemented,  no  requirements  creep  _ 


42|Structured  and  tracked  changes  to  requirements  onl 


43  Change  is  inevitable,  changes  allowed  at  all  times _ 


44  Change  is  inevitable,  but  changes  limited _ 


45 1  Requirements  control  funding _ _ _ 


Requirements  history  kept _ _ _ 


47  Baseline  established  for  requirements  at  some  point  prior  to  develop 
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Program  Name:  B 


YES-NO-N/A  Questionnaire 


Date:  Nov  99 


No.  Estimation/Planning  Questionnaire  _ Yes  No  N/A 


1 

A  volume  product  metric  used  (LOG,  #  of  files,  #  of  screens,  pages  of  doc) 

X 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

X 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOC  changed  at  unit  test) 

X 

4 

Other  product  attributes  measured  (FP.  throughput,  mem  cap,  cyclomatic  complexity) 

X 

5 

Product  metrics  tracked  and  updated  throughout  program  execution 

X 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

X 

7 

Time  measure  process  metric  used  (cycle  time) 

X 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

X 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

X 

10 

Program  cost  estimations  tracked  and  updated  to  reflect  progress/changes 

X 

11 

Factor  analysis  performed  on  program 

X 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

X 

13 

Work  breakdown  structure  developed 

X 

14 

Same  as  13,  but  management  oversees  and  formalizes 

X 

15 

Schedules  developed  based  on  realistic  expectations 

X 

16 

Schedules  tracked  and  updated  based  on  new  information 

X 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

X 

18 

Quality  assurance  plan  or  similar  to  aid  in  detecting  defects  early  in  program 

X 

19 

COCOMO  estimates  performed 

X 

20 

CSCI  defined  and  tasked 

X 

21 

Estimates  completed  ad  hoc 

X 

22 

Gantt  charts  used  and  updated 

X 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

X 

24 

Earned  value  established 

X 

25 

Earned  value  tracked  throughout  program 

X 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

X 

27 

Critical  path  for  program  tasks  developed  and  tracked 

X 

28 

Meaure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

X 

29 

Estimates  are  updated  routinely 

X 

30 

Schedules  are  updated  routinely 

X 

31 

Estimations  are  made  by  program  management  (top-down) 

X 

32 

Estimations  are  made  by  program  team  members  (bottom-up) 

X 

33 

Automated  program  tracking  used 

X 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

X 

35 

WBS  developed  only  as  data  call,  not  used  in  planning 

X 

36 

Earned  value  used  to  track  program  progress 

X 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

X 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

X 

39 

All  program  team  members  involved  in  planning  process 

X 

40 

Hardware  also  considered  in  estimation  process 

X 

41 

Program  history  compiled 

X 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

X 

43 

Management  duties  apart  of  each  team  member’s  responsibilities 

X 

44 

PM  dictates  schedules  to  program  team 

X 

45 

Code  reviews  planned  in  schedule 

X 

46 

Defined  tangible  milestones  established  for  program  tasks 

X 

47 

test  planning  done  at  the  start  of  the  program 

X 

48 

Estimations  are  completed  by  those  performing  the  tasks 

X 

49 

Sensitivity  analysis  performed  for  program  choices 

X 

50 

Software  deployment  planning  completed  prior  to  development  work 

ai 
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Program  Name:  B  YES-NO-N/A  Questionnaire 


No.  People  Management  Questionnaire _  Yes  No  N/A 


1 

PM  is  accessable  in  person  by  each  team  member 

X 

2 

PM  is  accessabie  via  emaii  by  each  team  member 

X 

3 

PM  is  accessable  via  phone  by  each  team  member 

X 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

X 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

X 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

X 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

X 

8 

PM  lets  teams  make  decisions  affecting  their  work 

X 

9 

PM  frequently  makes  decisions  without  any  consultation  with  members 

X 

10 

PM  understands  the  technology/language  of  the  program 

X 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

X 

12 

PM  prioritizes  problems  or  conflicts  within  the  program 

X 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

X 

14 

Same  as  13,  but  management  oversees  and  formalizes 

X 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

X 

16 

PM  specifically  assigns  work  to  each  program  member 

X 

17 

PM  sets  communication  protocol,  which  must  be  followed 

X 

18 

PM  allows  unrestricted  communications 

X 

19 

PM  readily  makes  tough  decisions 

X 

20 

PM  takes  control  in  difficult/  problem  areas 

X 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

X 

22 

PM  maintains  regular  communications  with  all  stakeholders 

X 

23 

PM  maintains  regular  communications  with  users 

X 

24 

PM  encourages  program  team  communication  with  users 

X 

25 

PM  encourages  program  team  communication  with  stakeholders 

X 

26 

PM  facilitates  horizontal  communication  within  program 

X 

27 

PM  facilitates  communication  during  integration 

X 

28 

PM  holds  meetings  without  objectives  listed  prior  to  meeting 

X 

29 

PM  must  approve  all  decisions  within  the  program 

X 

30 

PM  must  approve  all  interactions  with  stakeholders 

X 

31 

PM  must  approve  all  interactions  with  users 

X 

32 

PM  makes  all  presentations  to  stakeholders/users 

X 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

X 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

X 

35 

PM  is  readily  willing  to  listen  to  program  problems  and  complaints 

X 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

X 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

.X 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

X 

39 

PM  recruits  program  team  members  from  outside  organization 

X 

40 

PM  directs  what  needs  to  be  done  and  directs  how  to  do  it 

X 

41 

Program  personnel  have  clearly  defined  specific  tasks 

X 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

X 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

X 

44 

PM  delegation  of  duties  is  usually  seamless  in  execution 

X 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

X 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

X 

47 

PM  clearly  separates  technical  from  managerial  roles  for  individuals 

X 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

X 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

X 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

w 
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Program  Name:  B 


YES-NO-N/A  Questionnaire 


Date:  Nov  99 


No.  Risk  Management  Questionnaire _ Yes  No  N/A 


1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

X 

2 

RM  is  formal  and  documented 

X 

3 

A  specific  RM  plan  exists 

X 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

X 

5 

RM  is  done  prior  to  the  program  execution 

X 

6 

RM  is  done  by  an  outside  entity  to  the  development 

X 

7 

RM  is  done  internally  only 

X 

8 

RM  is  both  internally  performed  and  externally  assessed 

X 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

X 

10 

Risk  Assessment  is  only  a  management  function 

X 

11 

RM  is  informal  or  non  existent 

X 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

X 

13 

Risks  are  only  generalized 

X 

14 

Same  as  13,  but  management  oversees  and  formalizes 

X 

15 

Each  risk  has  a  consequence 

X 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

X 

17 

Each  risk  has  a  mitigation  strategy 

X 

18 

Risk  Management  is  automated 

X 

19 

Risks  are  tracked 

X 

21 

Regret  analysis  performed 

X 

22 

RM  drives  decisions  in  the  program 

X 

23 

Risks  have  probabilities 

X 

24 

Risk  Management  is  ad  hoc 

X 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

X 

26 

Risks  are  weighed  relative  to  other  program  risks 

X 

27 

Risk  Assessment  is  a  program  team  activity 

X 

28 

Risk  Assessment  done  prior  to  program  start 

X 

29 

Risk  Assessment  includes  personnel  risk 

X 

30 

RM  uses  tools,  but  depends  on  human  decisions 

X 

31 

Risk  Assessment  includes  cost  risks 

X 

32 

Risk  Assessment  includes  schedule  risks 

X 

33 

Risk  Assessment  includes  technology  risks 

X 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

X 

35 

Risk  Assessment  includes  requirements  risks 

X 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

X 

37 

Risk  Assessment  includes  documentation  risks 

X 

38 

Risk  Assessment  includes  integration  risks 

X 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

X 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

X 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

X 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

X 

43 

High  risk  have  measured  tracking  (high  profile  status) 

X 

44 

Organizational  history  used  to  search  for  risks 

X 

45 

Other  organizational  checklists  used  for  risk  assessment 

X 

46 

Internal  organizational  checklists  used  for  risk  assessment 

X 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

X 

48 

Risk  Assessment  includes  internal  organization  risks 

X 

49 

Risk  Assessment  includes  stakeholder  risks 

X 

50 

No  risk  management  needed:  program  is  straightforwarded  &  understood 
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Program  Name:  C  YES-NO-N/A  Questionnaire 

No.  Requirements  Management  Questionnaire _ 

1  |PM  chose  to  have  a  formai  requirements  list 
_ 2  Requirements  recorded  in  some  way _ _ 

3  Written  requirements  were  part  of  some  formal  document _ _ 

4  Written  requirements  were  informal _ _ 

5  At  least  some  requirements  were  oral  only _ _ 

6  All  stakeholders  were  identified _ 

7  All  stakeholders  participated  in  the  requirements  extraction _ 

8  Some  stakeholders  participated  in  the  requirements  extraction _ 

9  Management  extracted  requirements,  no  stakeholder  involvement _ 

10  Management  passed  requirements  to  development  team _ 

1 1  Stakeholders  not  involved  in  Management  extraction,  but  approves 

12  Management  gets  inputs  from  stakeholders,  then  develops  requirments" 

1 3  Developers  work  informally  with  users  to  arrive  at  requirements 
Isame  as  13.  but  management  oversees  and  formaliiii 


Date:  Nov  99 


Yes  No  N/A 


If  a  waterfall  or  sequential  development  strategy: _ _ 


All  requirements  complete  before  design _ 


Some  requirements  left  incomplete  prior  to  design _ 


Requirements  informal  prior  to  design  effort _ 


Requirements  serve  as  input _ 


Length  of  time  for  requirements  work  greater  than  development  work 


Requirements  developed  in  parallel  to  design 


OR  If  a  prototype,  throwaway,  or  other  development  strategy: _ 


Learn  about  requirements  through  development  efforts 


No  coding  until  all  requirements  are  defined _ _ 


Requirements  formal  prior  to  design  effort _ 


Requirements  serve  as  output _ 


Requirements  definition  work  in  parallel  to  development  efforts 


Requirements  developed  in  parallel  to  design 


Are  requirements  frozen  at  some  phase _ 


22|Change  management  exists _ 


23  Change  management  is  formal _ 


24  Project  strategy  is  consistent  throughout  development 


25 1  Requirements  are  updated _ _ 


Configuration  Management  (CM)  exists _ _ 


27 1  CM  is  formal  _ _ 


Requirements  are  testable _ 


Requirements  testing  considered/implemented  during  extraction 


Requirements  testing  plan  exists _ 


Requirements  testing  is  formal _ _ _ 


All  requirements  have  priorities _ _ _ 


33 1  All  requirements  must  be  implemented _ 


34  Requirements  are  tested _ _ 


35  All  requirements  are  equally  important _ _ _ 


At  least  some  requirements  have  priorities _ 


All  requirements  are  traceable _ 


Traceability  not  important _ _ 


Each  requirement  has  an  author _ _ 


Who  authored  requirement  is  not  Important _ 


41 1  Initial  set  of  requirements  to  be  implemented,  no  requirements  creep 


42 1  Structured  and  tracked  changes  to  requirements  only _ 


43|Change  is  inevitable,  changes  allowed  at  all  times _ 


44  Change  is  inevitable,  but  changes  limited _ 


45  Requirements  control  funding _ 


46  Requirements  history  kept _ _ _ 

47  Baseline  established  for  requirements  at  some  point  prior  to  develop _ 
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Program  Name:  C  YES-NO-N/A  Questionnaire  Date:  Nov  99 


No.  Estimation/Planning  Questionnaire  Yes  No  N/A 

1 

A  volume  product  metric  used  (LOG,  #  of  files,  #  of  screens,  pages  of  doc) 

X 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

X 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOG  changed  at  unit  test) 

X 

4 

Other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

X 

5 

Product  metrics  tracked  and  updated  throughout  program  execution 

X 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

X 

7 

Time  measure  process  metric  used  (cycle  time) 

X 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

X 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

X 

10 

Program  cost  estimations  tracked  and  updated  to  reflect  progress/changes 

X 

11 

Factor  analysis  performed  on  program 

X 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

X 

13 

Work  breakdown  structure  developed 

X 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

X 

15 

Schedules  developed  based  on  realistic  expectations 

X 

16 

Schedules  tracked  and  updated  based  on  new  information 

X 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

X 

18 

Quality  assurance  plan  or  similar  to  aid  in  detecting  defects  early  in  program 

X 

19 

COCOMO  estimates  performed 

X 

20 

CSCI  defined  and  tasked 

X 

21 

Estimates  completed  ad  hoc 

X 

22 

Gantt  charts  used  and  updated 

X 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

X 

24 

Earned  value  established 

X 

25 

Earned  value  tracked  throughout  program 

X 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

X 

27 

Critical  path  for  program  tasks  developed  and  tracked 

X 

28 

Meaure  of  effectiveness  (MQE)  or  Figure  of  merit  established  and  tracked 

X 

29 

Estimates  are  updated  routinely 

X 

30 

Schedules  are  updated  routinely 

X 

31 

Estimations  are  made  by  program  management  (top-down) 

X 

32 

Estimations  are  made  by  program  team  members  (bottom-up) 

X 

33 

Automated  program  tracking  used 

X 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

X 

35 

WBS  developed  only  as  data  call,  not  used  in  planning 

X 

36 

Earned  value  used  to  track  program  progress 

X 

37 

PM  Insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

X 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

X 

39 

All  program  team  members  involved  in  planning  process 

X 

40 

Hardware  also  considered  in  estimation  process 

X 

41 

Program  history  compiled 

X 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

X 

43 

Management  duties  apart  of  each  team  member’s  responsibilities 

X 

44 

PM  dictates  schedules  to  program  team 

X 

45 

Code  reviews  planned  in  schedule 

X 

46 

Defined  tangible  milestones  established  for  program  tasks 

X 

47 

Test  planning  done  at  the  start  of  the  program 

X 

48 

Estimations  are  completed  by  those  performing  the  tasks 

X 

49 

Sensitivity  analysis  performed  for  program  choices 

X 

50 

Software  deployment  planning  completed  prior  to  development  work 

X 

TOTAL  SCORING| 
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Program  Name:  C  YES-NO-N/A  Questionnaire 


No.  People  Management  Questionnaire  _ Yes  No  N/A 


1 

PM  is  accessable  in  person  by  each  team  member 

X 

2 

PM  is  accessable  via  email  by  each  team  member 

X 

3 

PM  is  accessable  via  phone  by  each  team  member 

X 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

X 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

X 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

X 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

X 

8 

PM  lets  teams  make  decisions  affecting  their  work 

X 

9 

PM  frequently  makes  decisions  without  any  consultation  with  members 

X 

10 

PM  understands  the  technology/language  of  the  program 

X 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

X 

12 

PM  prioritizes  problems  or  conflicts  within  the  program 

X 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

X 

14 

PM  empowers  program  members  to  recommend  hiring  of  other  members 

X 

15 

PM  empowers  program  members  to  recommend  firing  of  other  members 

X 

16 

PM  specifically  assigns  work  to  each  program  member 

X 

17 

PM  sets  communication  protocol  to  be  followed 

X 

18 

PM  allows  unrestricted  communications 

X 

19 

PM  readily  makes  tough  decisions 

X 

20 

PM  takes  control  in  difficult/  problem  areas 

X 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

X 

22 

PM  maintains  regular  communications  with  all  stakeholders 

X 

23 

PM  maintains  regular  communications  with  users 

X 

24 

PM  encourages  program  team  communication  with  users 

X 

25 

PM  encourages  program  team  communication  with  stakeholders 

X 

26 

PM  facilitates  horizontal  communication  within  program 

X 

27 

PM  facilitates  communication  during  integration 

X 

28 

PM  holds  meetings  without  objectives  listed  prior  to  meeting 

X 

29 

PM  must  approve  all  decisions  within  the  program 

X 

30 

PM  must  approve  all  interactions  with  stakeholders 

X 

31 

PM  must  approve  all  interactions  with  users 

X 

32 

PM  makes  all  presentations  to  stakeholders/users 

X 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

X 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

X 

35 

PM  is  readily  willing  to  listen  to  program  problems  and  complaints 

X 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

X 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

.  X 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

X 

39 

PM  recruits  program  team  members  from  outside  organization 

X 

40 

PM  directs  what  needs  to  be  done  and  directs  how  to  do  it 

X 

41 

Program  personnel  have  clearly  defined  specific  tasks 

X 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

X 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

X 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

X 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

X 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

X 

47 

PM  clearly  separates  technical  from  managerial  roles  for  individuals 

X 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

X 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

X 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

Bl 

^■1 

TOTAL  SCORINGI  421  61  o| 


Enter  total  score  on  QMM  score  sheet  block  g. 


Date:  Nov  99 


126 


Program  Name:  C 


YES-NO-N/A  Questionnaire 


Date:  Nov  99 


No.  Risk  Management  Questionnaire _ Yes  No  N/A 


1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

X 

2 

RM  is  formal  and  documented 

X 

3 

A  specific  RM  plan  exists 

X 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

X 

5 

RM  is  done  prior  to  the  program  execution 

X 

6 

RM  is  done  by  an  outside  entity  to  the  development 

X 

7 

RM  is  done  internally  only 

X 

8 

RM  is  both  internally  performed  and  externally  assessed 

X 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

X 

10 

Risk  Assessment  is  only  a  management  function 

X 

11 

RM  is  informal  or  non  existent 

X 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

X 

13 

Risks  are  only  generalized 

X 

14 

Each  risk  is  delineated 

X 

15 

Each  risk  has  a  consequence 

X 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

X 

17 

Each  risk  has  a  mitigation  strategy 

X 

18 

Risk  Management  is  automated 

X 

19 

Risks  are  tracked 

X 

21 

Regret  analysis  performed 

X 

22 

RM  drives  decisions  in  the  program 

X 

23 

Risks  have  probabilities 

X 

24 

Risk  Management  is  ad  hoc 

X 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

X 

26 

Risks  are  weighed  relative  to  other  program  risks 

X 

27 

Risk  Assessment  is  a  program  team  activity 

X 

28 

Risk  Assessment  done  prior  to  program  start 

X 

29 

Risk  Assessment  includes  personnel  risk 

X 

30 

RM  uses  tools,  but  depends  on  human  decisions 

X 

31 

Risk  Assessment  includes  cost  risks 

X 

32 

Risk  Assessment  includes  schedule  risks 

X 

33 

Risk  Assessment  includes  technology  risks 

X 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

X 

35 

Risk  Assessment  includes  requirements  risks 

X 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

X 

37 

Risk  Assessment  includes  documentation  risks 

X 

38 

Risk  Assessment  includes  integration  risks 

X 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

X 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

X 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

X 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

X 

43 

High  risk  have  measured  tracking  (high  profile  status) 

X 

44 

Organizational  history  used  to  search  for  risks 

X 

45 

Other  organizational  checklists  used  for  risk  assessment 

X 

46 

Internal  organizational  checklists  used  for  risk  assessment 

X 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

X 

48 

Risk  Assessment  includes  internal  organization  risks 

X 

49 

Risk  Assessment  includes  stakeholder  risks 

X 

50 

No  risk  management  needed;  program  is  straightforwarded  &  understood 

■1 
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APPENDIX  C 

FINAL  SURVEY  FORMS  TEMPLATE  WITH  SCORING 
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Program  Name _  YES-NO-N/A  Questionnaire  Scoring  Template 

No.  Requirements  Management  Questionnaire _ 

1 1  PM  chose  to  have  a  formal  requirements  list 

_ 2  Requirements  recorded  in  some  way _ 

_ 3  Written  requirements  were  part  of  some  formal  document _ 

_ 4  Written  requirements  were  informal _ 

5  At  least  some  requirements  were  oral  only _ 

_ 6  All  stakeholders  were  identified _ 

7  All  stakeholders  participated  in  the  requirements  extraction _ 

8  Some  stakeholders  participated  in  the  requirements  extraction _ 

9  Management  extracted  requirements,  no  stakeholder  involvement _ 

10  Management  passed  requirements  to  development  team _ 

1 1  Stakeholders  not  involved  in  Management  extraction,  but  approves _ 

12  Management  gets  inputs  from  stakeholders,  then  develops  requirments _ 

13  Developers  work  informally  with  users  to  arrive  at  requirements _ 

14  Same  as  13,  but  management  oversees  and  formalizes 


If  a  waterfall  or  sequential  development  strategy:  _ 


All  requirements  complete  before  design _ 


leiSome  requirements  left  incomplete  prior  to  design _ 


Requirements  informal  prior  to  design  effort _ 


Requirements  serve  as  input _ 


Length  of  time  for  requirements  work  greater  than  development  work  _ 


Requirements  developed  in  parallel  to  design 


OR  If  a  prototype,  throwaway,  or  other  development  strategy: _ 


Learn  about  requirements  through  development  efforts _ 


No  coding  until  all  requirements  are  defined _ 


Requirements  formal  prior  to  design  effort _ 


Requirements  serve  as  output _ 


Requirements  definition  work  in  parallel  to  development  efforts 


Requirements  developed  in  parallel  to  design 


Are  requirements  frozen  at  some  phase _ 


22 1  Change  management  exists _ 


23  Change  management  is  formal _ 


24 1  Project  strategy  is  consistent  throughout  development _ 


25  Requirements  are  updated _ 


26  Configuration  Management  (CM)  exists _ 


27  CM  is  formal  _ 


Requirements  are  testable _ 


29 1  Requirements  testing  considered/imp'lemented  during  extraction _ 


Requirements  testing  plan  exists _ 


31 1  Requirements  testing  is  formal _ 


32  All  requirements  have  priorities _ 


33  All  requirements  must  be  implemented _ 


34  Requirements  are  tested _ 


35  All  requirements  are  equally  important _ 


36  At  least  some  requirements  have  priorities _ 


37  All  requirements  are  traceable _ 


38|Traceability  not  important _ 


39 1  Each  requirement  has  an  author _ 


Who  authored  requirement  is  not  important _ 


41 1  Initial  set  of  requirements  to  be  implemented,  no  requirements  creep _ 


42  Structured  and  tracked  changes  to  requirements  onl 


43  Change  is  inevitable,  changes  allowed  at  all  times _ 


44  Change  is  inevitable,  but  changes  limited _ 


45 1  Requirements  control  funding _ 


Requirements  history  kept _ 


471  Baseline  established  for  requirements  at  some  point  prior  to  develop _ 


TOTAL  SCORING 
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Yes  No  N/A 

1  I  0  I  0~ 

_2 _ ^1 _ 0_ 

_J _ 0 _ 0_ 

_\ _ 2 _ 0_ 

-2  1  0 

_2 _ -J _ 0^ 

_2 _ 0 _ 0_ 

1  0  0 
^T"  2  1 

_J _ 0 _ 0_ 

_-1 _ 0 _ 0_ 

1  0  1 
1  0  0 

2  0  0 


1 

-3 

0 

-1 

0 

0 

-1 

0 

0 

1 

-1 

0 

2 

-1 

0 

-1 

1 

0 

11-10 


0 


-3  1 


-1  0 


1-10 


2  -10 


1-10 


1-10 


3  -3  0 


10  0 


10  0 


10  0 


3  -3  0 


10  0 


2  -2  0 


2  0  0 


2  0  10 


1  0 


2  -2 


0  10 


1-10 


0  10 


10  0 


10  0 


0  10 


10  0 


0  10 


0  10 


1-10 


-110 


10  0 


Program  Name _  YES-NO>N/A  Questionnaire  Scoring  Template  Date 


No.  Estimation/Planning  Questionnaire _ Yes  No  N/A 


1 

A  volume  product  metric  used  (LOG,  #  of  files,  #  of  screens,  pages  of  doc) 

1 

0 

0 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

1 

0 

0 

3 

P’roduct  measures  made  by  phase  (amt  at  implementation.  LOC  changed  at  unit  test) 

1 

0 

0 

4 

other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

1 

0 

0 

5 

Product  metrics  tracked  and  updated  throughout  program  execution 

2 

-1 

0 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

1 

0 

0 

7 

time  measure  process  metric  used  (cycle  time) 

1 

0 

0 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

2 

-1 

0 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

1 

0 

0 

10 

Program  cost  estimations  tracked  and  updated  to  reflect  progress/changes 

1 

0 

0 

11 

Factor  analysis  performed  on  program 

1 

0 

0 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

2 

-1 

0 

13 

Work  breakdown  structure  developed 

2 

-1 

0 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

1 

-1 

0 

15 

Schedules  developed  based  on  realistic  expectations 

1 

-1 

0 

16 

Schedules  tracked  and  updated  based  on  new  information 

1 

-1 

0 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

1 

-1 

0 

18 

Quality  assurance  plan  or  similar  to  aid  in  detecting  defects  early  in  program 

1 

-1 

0 

19 

COCOMO  estimates  performed 

1 

-1 

0 

20 

CSCI  clearly  defined  and  tasked 

2 

-1 

0 

21 

Estimates  completed  ad  hoc 

-2 

0 

0 

22 

CBantt  charts  used  and  updated 

1 

-1 

0 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

1 

-1 

0 

24 

Earned  value  established 

2 

-1 

0 

25 

Earned  value  tracked  throughout  program 

2 

0 

0 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

1 

-1 

0 

27 

Critical  path  for  program  tasks  developed  and  tracked 

2 

-1 

0 

28 

Meaure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

1 

0 

0 

29 

Estimates  are  updated  routinely 

2 

-1 

0 

30 

Schedules  are  updated  routinely 

2 

-1 

0 

31 

Estimations  are  made  by  program  management  (top-down) 

1 

0 

0 

32 

Estimations  are  made  by  program  team  members  (bottom-up) 

2 

0 

0 

33 

Automated  program  tracking  used 

1 

0 

0 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

1 

-1 

0 

35 

WBS  developed  only  as  data  call,  not  used  in  planning 

-1 

0 

0 

36 

Earned  value  used  to  track  program  progress 

2 

-1 

0 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

1 

-1 

0 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

2 

-1 

0 

39 

All  program  team  members  involved  in  planning  process 

2 

-1 

0 

40 

Hardware  also  considered  in  estimation  process 

1 

-1 

0 

41 

Program  history  compiled 

1 

0 

0 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

1 

-1 

0 

43 

Management  duties  apart  of  each  team  member's  responsibilities 

-1 

1 

0 

44 

PM  dictates  schedules  to  program  team 

-1 

0 

0 

45 

Code  reviews  planned  in  schedule 

1 

-1 

0 

46 

Defined  tangible  milestones  established  for  program  tasks 

2 

-1 

0 

47 

Test  planning  done  at  the  start  of  the  program 

1 

-1 

0 

48 

Estimations  are  completed  by  those  performing  the  tasks 

1 

-1 

0 

49 

Sensitivity  analysis  performed  for  program  choices 

1 

-1 

0 

50 

Software  deployment  planning  completed 

1 

-1 

0 

TOTAL  SCORINGI  |  j  j 
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Program  Name _  YES-NO-N/A  Questionnaire  Scoring  Template  Date 


No.  People  Management  Questionnaire _ Yes  No  N/A 


1 

PM  is  accessable  in  person  by  each  team  member 

1 

0 

0 

2 

PM  is  accessable  via  email  by  each  team  member 

1 

0 

0 

3 

PM  is  accessable  via  phone  by  each  team  member 

1 

0 

0 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

1 

0 

0 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

1 

0 

0 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

2 

-1 

0 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

2 

-1 

0 

8 

PM  lets  teams  make  decisions  affecting  their  work 

1 

0 

0 

9 

PM  frequently  makes  decisions  without  any  consultation  with  members 

-2 

2 

0 

10 

PM  understands  the  technology/language  of  the  program 

1 

0 

0 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

1 

-1 

0 

12 

PM  prioritizes  problems  or  conflicts  within  the  program 

1 

0 

0 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

1 

-1 

0 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

1 

-1 

0 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

1 

-1 

0 

16 

PM  specifically  assigns  work  to  each  program  member 

1 

-1 

0 

17 

PM  sets  communication  protocol  to  be  followed 

1 

0 

0 

18 

PM  allows  unrestricted  communications 

1 

0 

0 

19 

PM  readily  makes  tough  decisions 

1 

-1 

0 

20 

PM  takes  control  in  difficult/  problem  areas 

1 

0 

0 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

1 

0 

0 

22 

PM  maintains  regular  communications  with  all  stakeholders 

2 

-1 

0 

23 

PM  maintains  regular  communications  with  users 

2 

-1 

0 

24 

PM  encourages  program  team  communication  with  users 

1 

-1 

0 

25 

PM  encourages  program  team  communication  with  stakeholders 

1 

-1 

0 

26 

PM  facilitates  horizontal  communication  within  program 

1 

-1 

0 

27 

PM  facilitates  communication  during  integration 

1 

-1 

0 

28 

PM  holds  meetings  without  clear  objectives  listed  prior  to  meeting 

-1 

2 

!  0 

29 

PM  must  approve  all  decisions  within  the  program 

-1 

1 

0 

30 

PM  must  approve  all  interactions  with  stakeholders 

-1 

1 

0 

31 

PM  must  approve  all  interactions  with  users 

-1 

1 

0 

32 

PM  makes  all  presentations  to  stakeholders/users 

0 

1 

0 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

1 

0 

0 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

1 

0 

0 

35 

PM  is  readily  willing  to  listen  to  program  problems  and  complaints 

1 

-1 

0 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

1 

-1 

0 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

1 

-1 

0 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

-1 

1 

0 

39 

PM  recruits  program  team  members  from  outside  organization 

1 

-1 

0 

40 

PM  directs  what  needs  to  be  done  and  directs  how  to  do  it 

-1 

1 

0 

41 

Program  personnel  have  clearly  defined  specific  tasks 

0 

1 

0 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

2 

-1 

0 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

2 

-1 

0 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

1 

0 

0 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

2 

-1 

0 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

2 

-1 

0 

47 

PM  clearly  separates  technical  from  managerial  roles  for  individuals 

0 

1 

0 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

0 

1 

0 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

2 

-1 

0 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

2 

-1 

0 

TOTAL  SCORING!  |  |  | 


Enter  total  score  on  QMM  score  sheet  block  g. 
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Program  Name _  YES-NO-N/A  Questionnaire  Scoring  Template  Date 


No.  Risk  Management  Questionnaire _ Yes  No  N/A 


1 

Risk  Management  (RM)  is  specificaily  an  activity  in  the  program 

4 

-4 

0 

2 

RM  is  formal  and  documented 

3 

-3 

0 

3 

A  specific  RM  plan  exists 

2 

-2 

0 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

-1 

1 

0 

5 

RM  is  done  prior  to  the  program  execution 

1 

0 

0 

6 

RM  is  done  by  an  outside  entity  to  the  development 

1 

0 

0 

7 

RM  is  done  internally  only 

0 

1 

0 

8 

RM  is  both  internally  performed  and  externally  assessed 

1 

-1 

0 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

1 

-1 

0 

10 

Risk  Assessment  is  only  a  management  function 

0 

1 

0 

11 

RM  is  informal  or  non  existent 

-1 

1 

0 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

1 

0 

0 

13 

Risks  are  only  generalized 

-1 

0 

0 

14 

Each  risk  is  delineated 

1 

0 

0 

15 

Each  risk  has  a  consequence 

1 

0 

0 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

1 

0 

0 

17 

Each  risk  has  a  mitigation  strategy 

1 

0 

0 

18 

Risk  Management  is  automated 

1 

0 

0 

19 

Risks  are  tracked 

2 

-2 

0 

21 

Regret  analysis  performed 

2 

0 

0 

22 

RM  drives  decisions  in  the  program 

3 

-2 

0 

23 

Risks  have  probabilities 

1 

0 

0 

24 

Risk  Management  is  ad  hoc 

-3 

0 

0 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

1 

0 

0 

26 

Risks  are  weighed  relative  to  other  program  risks 

1 

0 

0 

27 

Risk  Assessment  is  a  program  team  activity 

1 

0 

0 

28 

Risk  Assessment  done  prior  to  program  start 

2 

-1 

0 

29 

Risk  Assessment  includes  personnel  risk 

1 

-1 

0 

30 

RM  uses  tools,  but  depends  on  human  decisions 

2 

-1 

0 

31 

Risk  Assessment  includes  cost  risks 

1 

0 

0 

32 

Risk  Assessment  includes  schedule  risks 

1 

0 

0 

33 

Risk  Assessment  includes  technology  risks 

1 

-1 

0 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

1 

-1 

0 

35 

Risk  Assessment  includes  requirements  risks 

1 

-1 

0 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

1 

0 

0 

37 

Risk  Assessment  includes  documentation  risks 

1 

0 

0 

38 

Risk  Assessment  includes  integration  risks 

1 

-1 

0 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

1 

-1 

0 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

1 

-1 

0 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

1 

0 

0 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

1 

0 

0 

43 

High  risk  have  measured  tracking  (high  profile  status) 

1 

0 

0 

44 

Organizational  history  used  to  search  for  risks 

1 

0 

0 

45 

Other  organizational  checklists  used  for  risk  assessment 

1 

0 

0 

46 

Internal  organizational  checklists  used  for  risk  assessment 

1 

0 

0 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

1 

0 

0 

48 

Risk  Assessment  includes  internal  organization  risks 

1 

0 

0 

49 

Risk  Assessment  includes  stakeholder  risks 

2 

-1 

0 

50 

No  risk  management  needed;  program  is  straightforwarded  &  understood 

-3 

3 

0 

TOTAL  scoring!  I  II 


Enter  total  score  on  QMM  score  sheet  block  h. 
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